Class: Downloader

(package) downloader.Downloader(dataSource)

Use a Downloader object to download a set of URLs at regular time intervals.

This class is used by module:synchronizer.synchronizer and module:notifications.notifications.

Constructor

(package) new Downloader(dataSource)

Creates a new downloader instance.

Parameters:
Name Type Description
dataSource module:downloader~DataSource

A function that yields module:downloader.Downloadable objects on each check.

Source:
Example
function* dataSource()
{
  yield new Downloadable("https://example.com/filters.txt");
}

let initialDelay = 1000;
let checkInterval = 60000;

let downloader = new Downloader(dataSource);
downloader.scheduleChecks(checkInterval, initialDelay);

downloader.onDownloadStarted = function(downloadable)
{
  console.log(`Downloading ${downloadable.url} ...`);
}

Members

_downloading :Set.<string>

Set containing the URLs of objects currently being downloaded.

Type:
  • Set.<string>
Source:

_timeout :number

Store the timeout ID for scheduled checks.

Type:
  • number
Source:

dataSource :module:downloader~DataSource

A function that yields module:downloader.Downloadable objects on each check.

Type:
Source:

maxAbsenceInterval :number

Maximal time interval that the checks can be left out until the soft expiration interval increases.

Type:
  • number
Source:

maxExpirationInterval :number

Maximal allowed expiration interval; larger expiration intervals will be corrected.

Type:
  • number
Source:

maxRedirects :number

Maximal number of redirects before the download is considered as failed.

Type:
  • number
Source:

minRetryInterval :number

Minimal time interval before retrying a download after an error.

Type:
  • number
Source:

(nullable) onDownloadError :function

Callback to be triggered whenever a download fails.

Type:
  • function
Source:

(nullable) onDownloadStarted :function

Callback to be triggered whenever a download starts.

Type:
  • function
Source:

(nullable) onDownloadSuccess :function

Callback to be triggered whenever a download finishes successfully.

The callback can return an error code to indicate that the data is wrong.

Type:
  • function
Source:

(nullable) onExpirationChange :function

Called whenever expiration intervals for an object need to be adapted.

Type:
  • function
Source:

Methods

(package, static) isValidURL(url) → {boolean}

Checks whether a URL is a valid download URL.

For a URL to be a valid download URL, its scheme must be https or data. If the host component of the URL is localhost, 127.0.0.1, or [::1], however, but its scheme is not https or data, the URL is still considered to be a valid download URL.

https://example.com/, data:,Hello%2C%20World!, and http://127.0.0.1/example.txt are all examples of valid download URLs.

Parameters:
Name Type Description
url string

The URL.

Source:
Returns:

Whether the URL is a valid download URL.

Type
boolean

_doCheck()

Checks whether anything needs downloading.

Source:

download(downloadable)

Starts downloading for an object.

Parameters:
Name Type Description
downloadable Downloadable
Source:

getDownloadUrl(downloadable) → {string}

Generates the real download URL for an object by appending various parameters.

Parameters:
Name Type Description
downloadable Downloadable
Source:
Returns:
Type
string

isDownloading(url) → {boolean}

Checks whether an address is currently being downloaded.

Parameters:
Name Type Description
url string
Source:
Returns:
Type
boolean

processExpirationInterval(interval) → {Array.<number>}

Produces a soft and a hard expiration time for a given supplied expiration interval.

Downloadable things, like Subscription, have a certain time after which they expire and you need to download a new, updated version.

To spread the load on our download servers, we'd rather that all users not download updates at exactly the same time, so on download we calculate a soft expiration time, which is randomly chosen between 80% and 120% of the original time. When we pass the soft expiration, we start trying to download updates.

However, there are other events that could cause downloads to all happen at the same time. If many people leave their computer off over the weekend and sign in at the beginning of business hours on Monday, we could still see a spike in downloads at the same time. To combat this, the soft expiration time may be delayed to accomodate these periods of inactivity.

There are limits to how far we want to delay our download due to inactivity. This is why we also calculate a hard expiration time, which is 200% of the original time. When we pass the hard expiration, we should try to download an update whether there was inactivity or not.

This function must be called to calculate the new expiration times as soon as a new version is downloaded.

Parameters:
Name Type Description
interval number

how often in milliseconds the downloadable should be updated

Source:
Returns:

soft and hard expiration times in milliseconds since the Unix epoch

Type
Array.<number>

scheduleChecks(interval, delay)

Schedules checks at regular time intervals.

Parameters:
Name Type Description
interval number

The interval between checks in milliseconds.

delay number

The delay before the initial check in milliseconds.

Source:

unscheduleChecks()

Clear previously scheduled checks, if any.

Source: