Categories
discuss

Concurrency limit in Q promises – node

Is there any method to limit concurrency of promises using Q promises library?

This question is kinda related to How can I limit Q promise concurrency?

but the problem is that I’m trying to do something like this:

for (var i = 0; i <= 1000; i++) {
  return Q.all([ task1(i), task2(i) ]); // <-- limit this to 2 at a time.
}

The real use case is:

  1. Fetch posts from DB
  2. Loop every post in DB like posts.forEach(function(post) {}
  3. For every post do task1, task2, task3 (retrieve social counters, retrieve comments count, etc)
  4. Save new post data in DB.

But the problem is that node is executing all tasks for all posts at the same time, like asking facebook for the “likes count” for 500 posts at the same time.

How i can limit Q.all() so only 2 posts at a time are executing their tasks? Or what other possible solutions can apply here?

Note: Most of the tasks (if not all) rely on request library

Answer

Thanks to Dan, his answer and his help to integrate it with my code, it can be done using his gist and a snipplet like this:

var qlimit = require('../libs/qlimit');

var test = function(id) {
  console.log('Running ' + id);
  return Q.nfcall(request, 'some dummy url which takes some time to process, for example a php file with sleep(5)').spread(function(response, body) {
    console.log('Response ' + id);
    return body;
  });
}

test = qlimit.limitConcurrency(test, 1);

var data = [0, 1, 2];

data.forEach(function(id) {
  console.log('Starting item ' + id);
  Q.all([ test(id) ]);
});

This way you get something like:

  • Starting item 0
  • Starting item 1
  • Starting item 2
  • Running 0
  • Response 0
  • Running 1
  • Response 1
  • Running 2
  • Response 2

Which clearly is 1 request at a time.

The whole point that i was missing in the implementation is that you need to re-declare the function using limitConcurrency BEFORE starting the loop, not inside it.

Categories
discuss

Why does using one “print” instead of three work?

So I don’t know much about java but I noticed this worked then according to my class notes I should be doing it a different way

Here’s what my notes have

System.out.print("hello");
System.out.print(name);
System.out.print("n");

However I tried this and it also does the same thing. It’s shorter so is this an acceptable way to do it or will it break down the road?

System.out.print("hello"+name+"n);

Also as long as the code runs right my teacher shouldn’t care right? It’s my first class and I’m not a computer science major.

Answer

It will work and I’d argue that it’s in fact a better way to do it.

Go for it, that’s the hacker spirit!

If you want something even shorter and more descriptive, try

System.out.println("hello " + name);

The println will automatically print a line end (‘n’) at the end of what you print.


Just to make this complete, let’s assume name = "James Gosling";.

In the code written in your notes, you first print:

hello

Then, you print name, which leads to:

helloJames Gosling

It’s printed like that because we’re actually missing a space after "hello". To print it with a space, use "hello ". Finally, we print a newline.

In your (arguably better) piece of code, you print only once, but when you use the expression "hello"+name+"n", you are creating a new character string which ends up being the same. This is because the + operator concatenates (that is, chains) two strings and creates a new string with the result.

So, when you print the resulting string, you get (plus the newline):

helloJames Gosling

Categories
discuss

Collecting android sensor for a specific period and calculating the average

I am trying to write a method that collects accelerometer sensor values over a specific time period and returns the average of the sensor readings for that period.

It should be a synchronous i.e. blocking method that once it is called will block the calling thread for sometime and then will return the sensor average

I did check the below similar questions but does not seem to have a proper working solution for my case:

SensorEventListener in separate thread

Android – how to run your sensor ( service, thread, activity )?

Android sensors and thread

A method for waiting for sensor data

I’ve also tried to use Executors similar to this question, but could not get it to work as I want.

Below is my code skeleton, where the method sensorAverage is a blocking method that will calculate the accelerometer sensor average over a period equals to the timeout parameter

Average average = new Average(); // Some class to calculate the mean

double sensorAverage(long timeout){
    Sensor sensor = sensorManager.getDefaultSensor(Sensor.TYPE_LINEAR_ACCELERATION);
    sensorManager.registerListener(this, sensor,SensorManager.SENSOR_DELAY_NORMAL);

    // This does not work
    Thread.sleep(timeout);

    sensorManager.unregisterListener(this);

    return average.value();
}

public void onSensorChanged(SensorEvent event) {
    if (event.sensor.getType() == Sensor.TYPE_LINEAR_ACCELERATION) {
        double x2 = Math.pow(event.values[0], 2);
        double y2 = Math.pow(event.values[1], 2);
        double z2 = Math.pow(event.values[2], 2);
        average.add(Math.sqrt((x2 + y2 + z2)));
    }
}

Edit: I am aware that I need another thread, but the problem that I need to run it for a specific period only and so far I cannot find a proper working solution. Because when I use another thread I get the sensor average always 0

Answer

I managed to implement a solution which exactly does what I want.

A blocking method that collects sensor values for specific period and returns the statistics of all sensor readings i.e. mean and variance.

It is possible to simply store all the sensor’s values and then calculate the mean and variance; however you might run out of memory in case of collecting high frequency sensor over extended period of time.

I found a better solution to calculate the mean and variance for a stream of data in real-time (i.e. without storing the sensor values) using the below RunningStat class

Example code:

// Calculate statistics of accelerometer values over 300 ms (a blocking method)
RunningStat[] stats = SensorUtils.sensorStats(context, 
                                                  Sensor.TYPE_ACCELEROMETER, 300)
double xMean = stats[0].mean();
double xVar  = stats[0].variance();

Full class code:

public class SensorUtils {

// Collect sensors data for specific period and return statistics of 
// sensor values e.g. mean and variance for x, y and z-axis
public static RunningStat[] sensorStats(Context context, int sensorType,
        long timeout) throws Exception {
    ExecutorService executor = Executors.newSingleThreadExecutor();
    Future<RunningStat[]> future = executor.submit(new SensorTask(context,
            sensorType, timeout));

    RunningStat[] stats = future.get();
    return stats;

}

private static class SensorTask implements Callable<RunningStat[]> {
    private final Context context;
    private final long timeout;
    private final int sensorType;
    // We need a dedicated handler for the onSensorChanged
    HandlerThread handler = new HandlerThread("SensorHandlerThread");

    public SensorTask(Context context, int sensorType, long timeout) {
        this.context = context;
        this.timeout = timeout;
        this.sensorType = sensorType;
    }

    @Override
    public RunningStat[] call() throws Exception {
        final SensorCollector collector = new SensorCollector(context);
        handler.start();
        Thread sensorThread = new Thread() {
            public void run() {
                collector.start(sensorType,
                        new Handler(handler.getLooper()));
            };
        };
        sensorThread.start();
        Thread.sleep(timeout);
        return collector.finishWithResult();
    }
}

private static class SensorCollector implements SensorEventListener {
    protected Context context;
    protected RunningStat[] runningStat;
    protected SensorManager sensorManager;
    protected int sensorType;

    public SensorCollector(Context context) {
        this.context = context;
    }

    protected void start(int sensorType, Handler handle) {
        if (runningStat == null) {
            runningStat = new RunningStat[3];
            runningStat[0] = new RunningStat(3);
            runningStat[1] = new RunningStat(3);
            runningStat[2] = new RunningStat(3);
        } else {
            runningStat[0].clear();
            runningStat[1].clear();
            runningStat[2].clear();
        }
        this.sensorType = sensorType;
        sensorManager = (SensorManager) context
                .getSystemService(Context.SENSOR_SERVICE);
        Sensor sensor = sensorManager.getDefaultSensor(sensorType);
        sensorManager.registerListener(this, sensor,
                SensorManager.SENSOR_DELAY_NORMAL, handle);
    }

    public RunningStat[] finishWithResult() {
        if (sensorManager != null) {
            sensorManager.unregisterListener(this);
        }
        return runningStat;
    }

    @Override
    public void onAccuracyChanged(Sensor sensor, int accuracy) {

    }

    @Override
    public void onSensorChanged(SensorEvent event) {
        if (event.sensor.getType() == sensorType) {
            runningStat[0].push(event.values[0]);
            runningStat[1].push(event.values[1]);
            runningStat[2].push(event.values[2]);
        }
    }
}

}

Here is the RunningStat code, which is a very handy class to calculate the mean and variance for stream of data without storing the data itself (perfect for calculating statistics of high frequency sensors with very small memory footprint)

//See Knuth TAOCP vol 2, 3rd edition, page 232
public class RunningStat {
private int n;
private double oldM, newM, oldS, newS;
private int precision = -1;

// An estimate for the t-value (can be read from the t-distribution table)
private static final double T_THRESHOLD = 1.68;

public RunningStat(int precision) {
    this.precision = precision;
}

public RunningStat() {
}

public void clear() {
    n = 0;
}

public void push(double x) {
    n++;


    if (n == 1) {
        oldM = newM = x;
        oldS = 0.0;
    } else {
        newM = oldM + (x - oldM) / n;
        newS = oldS + (x - oldM) * (x - newM);

        // set up for next iteration
        oldM = newM;
        oldS = newS;
    }
}

public int count() {
    return n;
}

public double mean() {
    double mean = (n > 0) ? newM : 0.0;
    if (precision > 0) {
        return round(mean, precision);
    }
    return mean;
}

// The upper bound of the mean confidence interval
public double meanUpper() {
    double mean = (n > 0) ? newM : 0.0;
    double stdError = stdDeviation() / Math.sqrt(n);
    double upperMean = mean + T_THRESHOLD * stdError;
    if (precision > 0) {
        return round((n > 0) ? upperMean : 0.0, precision);
    }
    return upperMean;
}

// The lower bound of the mean confidence interval
public double meanLower() {
    double mean = (n > 0) ? newM : 0.0;
    double stdError = stdDeviation() / Math.sqrt(n);
    double lowerMean = mean - T_THRESHOLD * stdError;
    if (precision > 0) {
        return round((n > 0) ? lowerMean : 0.0, precision);
    }
    return lowerMean;
}

public double variance() {
    if (precision > 0) {
        return round(((n > 1) ? newS / (n - 1) : 0.0), precision);
    }
    return ((n > 1) ? newS / (n - 1) : 0.0);
}

public double stdDeviation() {
    if (precision > 0) {
        return round(Math.sqrt(variance()), precision);
    }
    return Math.sqrt(variance());
}

public void setPrecision(int precision) {
    this.precision = precision;
}

    public static double round(double value, int precision) {
         BigDecimal num = new BigDecimal(value);
         num = num.round(new MathContext(precision, RoundingMode.HALF_UP));
         return num.doubleValue();
    }

// A small test case
public static void main(String[] args) {
    int n = 100;
    RunningStat runningStat = new RunningStat();
    double[] data = new double[n];
    double sum = 0.0;
    for (int i = 0; i < n; i++) {
        data[i] = i * i;
        sum += data[i];
        runningStat.push(data[i]);
        System.out.println(runningStat.mean() + " - "
                + runningStat.variance() + " - "
                + runningStat.stdDeviation());
    }

    double mean = sum / n;
    double sum2 = 0.0;

    for (int i = 0; i < n; i++) {
        sum2 = sum2 + (data[i] - mean) * (data[i] - mean);
    }

    double variance = sum2 / (n - 1);
    System.out.println("nn" + mean + " - " + variance + " - "
            + Math.sqrt(variance));
}
}
Categories
discuss

Simulate drop file event

Is it possible to simulate/fake the drop event using javascript only? How to test this type of event?

Take for example this dnd upload sample page , is it possible to trigger the “drop” event with a file without actually dropping a file there? Let’s say clicking on a button?

I have started writing a Sukuli script that can control the mouse and do the trick but I was looking for a better solution.

EDIT

@kol answer is a good way to get rid of the drag and drop event but I still have to manually select a file from my computer. This is the bit I am interested in simulating. Is there a way to create a file variable programatically?

var fileInput = document.getElementById('fileInput'),
file = fileInput.files[0];    

Answer

1. Dropping image selected by the user

I’ve made a jsfiddle. It’s a stripped-down version of the html5demos.com page you’ve referred to, but:

  • I added an <input type="file"> tag which can be used to select an image file from the local computer, and
  • I also added an <input type="button"> tag with an onclick handler, which simulates the “drop file” event by directly calling the ondrop event handler of the DND-target div tag.

The ondrop handler looks like this:

holder.ondrop = function (e) {
    this.className = '';
    e.preventDefault();
    readfiles(e.dataTransfer.files);
}

That is, we have to pass an argument to ondrop, which

  • has a dataTransfer field with a files array subfield, which contains the selected File, and
  • has a preventDefault method (a function with no body will do).

So the onclick handler of the “Simulate drop” button is the following:

function simulateDrop() {
    var fileInput = document.getElementById('fileInput'),
        file = fileInput.files[0];        
    holder.ondrop({ 
        dataTransfer: { files: [ file ] }, 
        preventDefault: function () {} 
    });
}

Test

  1. Select an image file (png, jpeg, or gif)
  2. Click on the “Simulate drop” button

Result

Result

2. Dropping autogenerated test files without user interaction (GOOGLE CHROME ONLY!!!)

I’ve made another jsfiddle. When the page is loaded, a function gets called, which:

  • creates a text file into the temporary file system, and
  • loads and drops this text file into the target <div>; then
  • creates an image file into the temporary file system, and
  • loads and drops this image file into the target <div>.

The code of this drop-simulator function call is the following:

(function () {
    var fileErrorHandler = function (e) {
            var msg = "";
            switch (e.code) {
                case FileError.QUOTA_EXCEEDED_ERR:
                    msg = "QUOTA_EXCEEDED_ERR";
                    break;
                case FileError.NOT_FOUND_ERR:
                    msg = "NOT_FOUND_ERR";
                    break;
                case FileError.SECURITY_ERR:
                    msg = "SECURITY_ERR";
                    break;
                case FileError.INVALID_MODIFICATION_ERR:
                    msg = "INVALID_MODIFICATION_ERR";
                    break;
                case FileError.INVALID_STATE_ERR:
                    msg = "INVALID_STATE_ERR";
                    break;
                default:
                    msg = "Unknown Error";
                    break;
            };
            console.log("Error: " + msg);
        },
        requestFileSystem = window.requestFileSystem || window.webkitRequestFileSystem,
        dropFile = function (file) {
            holder.ondrop({ 
                dataTransfer: { files: [ file ] }, 
                preventDefault: function () {} 
            });
        };

    if (!requestFileSystem) {
        console.log("FileSystem API is not supported");
        return;
    }
    requestFileSystem(
        window.TEMPORARY, 
        1024 * 1024, 
        function (fileSystem) {
            var textFile = {
                    name: "test.txt",
                    content: "hello, world",
                    contentType: "text/plain"
                },
                imageFile = {
                    name: "test.png",
                    content: "iVBORw0KGgoAAAANSUhEUgAAAAUAAAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO9TXL0Y4OHwAAAABJRU5ErkJggg==",
                    contentType: "image/png",
                    contentBytes: function () {
                        var byteCharacters = atob(this.content),
                            byteArrays = [], offset, sliceSize = 512, slice, byteNumbers, i, byteArray;

                        for (offset = 0; offset < byteCharacters.length; offset += sliceSize) {
                            slice = byteCharacters.slice(offset, offset + sliceSize);
                            byteNumbers = new Array(slice.length);
                            for (i = 0; i < slice.length; i++) {
                                byteNumbers[i] = slice.charCodeAt(i);
                            }
                            byteArray = new Uint8Array(byteNumbers);
                            byteArrays.push(byteArray);
                        }
                        return byteArrays;
                    }
                };

            // Create and drop text file
            fileSystem.root.getFile(
                textFile.name, 
                { create: true }, 
                function (fileEntry) {
                    fileEntry.createWriter(
                        function (fileWriter) {
                            fileWriter.onwriteend = function(e) {
                                console.log("Write completed (" + textFile.name + ")");
                                fileSystem.root.getFile(
                                    textFile.name, 
                                    {}, 
                                    function (fileEntry) {
                                        fileEntry.file(
                                            function (file) {
                                                dropFile(file);
                                            }, 
                                            fileErrorHandler
                                        );
                                    }, 
                                    fileErrorHandler
                                );    

                            };
                            fileWriter.onerror = function(e) {
                                console.log("Write failed (" + textFile.name + "): " + e.toString());
                            };
                            fileWriter.write(new Blob([ textFile.content ], { type: textFile.contentType }));
                        }, 
                        fileErrorHandler
                    );
                }, 
                fileErrorHandler
            );

            // Create and drop image file
            fileSystem.root.getFile(
                imageFile.name, 
                { create: true }, 
                function (fileEntry) {
                    fileEntry.createWriter(
                        function (fileWriter) {
                            fileWriter.onwriteend = function(e) {
                                console.log("Write completed (" + imageFile.name + ")");
                                fileSystem.root.getFile(
                                    imageFile.name, 
                                    {}, 
                                    function (fileEntry) {
                                        fileEntry.file(
                                            function (file) {
                                                dropFile(file);
                                            }, 
                                            fileErrorHandler
                                        );
                                    }, 
                                    fileErrorHandler
                                );    

                            };
                            fileWriter.onerror = function(e) {
                                console.log("Write failed (" + imageFile.name + "): " + e.toString());
                            };
                            fileWriter.write(new Blob(imageFile.contentBytes(), { type: imageFile.contentType }));
                        }, 
                        fileErrorHandler
                    );
                }, 
                fileErrorHandler
            );
        }, 
        fileErrorHandler
    );    
})();

The content of the auto-generated text file is given as a string, and the content of the image file is given as a base64-encoded string. These are easy to change. For example, the test text file can contain not just plain text, but HTML too. In this case, don’t forget to change the textFile.contentType field from text/plain to text/html, and to add this content type to the acceptedTypes array and to the previewfile function. The test image can also be changed easily, you just need an image-to-base64 converter.

I had to extend the drop handler code to handle text files in addition to images:

acceptedTypes = {
    'text/plain': true, // <-- I added this
    'image/png': true,
    'image/jpeg': true,
    'image/gif': true
},

...

function previewfile(file) {
    if (tests.filereader === true && acceptedTypes[file.type] === true) {
        var reader = new FileReader();
        if (file.type === 'text/plain') { // <-- I added this branch
            reader.onload = function (event) {
                var p = document.createElement("p"); 
                p.innerText = event.target.result;
                holder.appendChild(p);
            }
            reader.readAsText(file);
        } else {
            reader.onload = function (event) {
                var image = new Image();
                image.src = event.target.result;
                image.width = 250; // a fake resize
                holder.appendChild(image);
            };
            reader.readAsDataURL(file);
        }
    } else {
        holder.innerHTML += '<p>Uploaded ' + file.name + ', ' + file.size + ' B, ' + file.type;
        console.log(file);
    }
}

Note that after loading the jsfiddle, the autogenerated files can be listed for debugging purposes:

Temporary file system

Result

Result

The screenshot shows that the simulated drop inserted the content of the autogenerated text file before the autogenerated image. The HTML code of the DND-target <div> looks like this:

<div id="holder" class="">
    <p>hello, world</p>
    <img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUAAAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO9TXL0Y4OHwAAAABJRU5ErkJggkFBTzlUWEwwWTRPSHdBQUFBQkpSVTVFcmtKZ2dnPT0=" width="250">
</div>
Categories
discuss

Launching Instance: VPC security groups may not be used for a non-VPC launch

I’m attempting to create an instance in another region, but I get this error:

AWS Error Code: InvalidParameterCombination, AWS Error Message: VPC security groups may not be used for a non-VPC launch

Here is the code I’m executing.

RunInstancesRequest instancereq = new RunInstancesRequest();

instancereq.setInstanceType("m3.medium");
instancereq.setImageId("ami-37b1b45e");
instancereq.setMinCount(1);
instancereq.setMaxCount(1);

ArrayList<String> secgroup = new ArrayList<String>();        
instancereq.setKeyName("testkey");          
secgroup.add("testdefault");          
instancereq.setSecurityGroups(secgroup);

instancereq.setPlacement(getAzPlacement());        
RunInstancesResult instanceresult = ec2.runInstances(instancereq);

I’ve also tried, instead of using the name “testdefault”, using the actual groupid (sg-########), but I’ll get an error saying that security group doesn’t exist (which is wrong, it does). Which, based on the API doc, if using a non-default VPC, you should pass the actual groupid but I’ll get an error like this:

InvalidGroup.NotFound, AWS Error Message: The security group 'sg-########' does not exist

If I use “default” as the setSecurityGroups it will use the default VPC. It just doesn’t seem like like the groupid I’m passing, despite it being accurate.

Also, if I comment out the setSecurityGroups code, and use setSubnetId instead and pass the subnet id, it will create the instance just fine, but it goes into the “default” security group, not “testdefault” like I want.

All I’m trying to accomplish is creating an instance and having it use the already existing VPC group.

Answer

My Answer will focus on below statement:

All I’m trying to accomplish is creating an instance and having it use the already existing VPC group.

So, as I understand, you want to launch an instance in a non-default VPC and assign it an existing VPC security group to it.

I am not a java guy, but I could do what you wanted in ruby as below.

require 'aws-sdk-core'
Aws.config = {
  :access_key_id => "my_access_key",
  :secret_access_key => "my_secret_key",
  :region => 'us-west-2'
}

ec2 = Aws::EC2.new

ec2.run_instances(
    min_count: 1,
    max_count: 1,
    image_id: 'ami-8635a9b6',
    instance_type: 't1.micro',
    placement: {
      availability_zone: 'us-west-2a'
    },
    network_interfaces: [
      {
        subnet_id: 'subnet-e881bd63',
        groups: ['sg-fd53bf5e'],
        device_index: 0,
        associate_public_ip_address: true
      }
    ],
    key_name: 'my-key'
).each do |resp|
  resp.instances.each do |x|
    puts x.instance_id
  end
end

Although this is a Ruby code, it is pretty straight forward and should give you some clear hints on how to go about doing it in Java as all these AWS SDKs are polling the same web service APIs.

I guess, the things that you should be concentrating in above code is:

  :region => 'us-west-2'

and

placement: {
  availability_zone: 'us-west-2a'
},
network_interfaces: [
  {
    subnet_id: 'subnet-e881bd63',
    groups: ['sg-fd53bf5e'],
    device_index: 0,
    associate_public_ip_address: true
  }
],
  1. Make sure you explicitly specify the region.
  2. Check how I have defined the subnet ID and security group ID. This code will launch my EC2 instance in subnet-e881bd63 of my VPC and will apply VPC security group ID sg-fd53bf5e to its 0th network interface. Besides, it will also assign a public IP address to my instance. (by default, it will not assign a public IP address when you launch instances in VPC).
  3. FYI. When you launch instances in VPC, you must provide Security group ID instead of security group name.
Source: stackoverflow
Text is available under the Creative Commons Attribution-ShareAlike License; additional terms may apply. By using this site, you agree to the Privacy Policy, and Copyright Policy. Content is available under CC BY-SA 3.0 unless otherwise noted. The answers/resolutions are collected from stackoverflow, are licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0 © No Copyrights, All Questions are retrived from public domain..