CHANGELOG.md

# 0.44.0 (2024-09-06)

* Fix checking of ctime/mtime of already imported files (in the importer)
* The autoimporter now correctly adds ugoira videos (not just the zips) to the import queue
* Updated gallery-dl to latest version

# 0.43.0 (2024-08-13)

* Updated gallery-dl and yt-dlp to latest versions
* Removed the long dead youtube-dl from dependencies and fixed the default config that was still preferring it over yt-dlp
* Fixed a bug in the importer introduced in 0.42.0 where an empty `additional_data` field would get parsed as the "None" string instead of an empty string
* Importer now skips log cleanup on boot when `--ignore-daemon` is used to avoid a crash (but you should NOT use `--ignore-daemon`)
* Import rules for AIBooru added to the default importer config (contributed by a user)
* The default importer configuration gained a new `hydl-import-time` tag that records the start time of the import job (so if multiple files were imported in the same run they will have the same value)
* Documentation updates

# 0.42.0 (2024-07-19)

* Fixed an error when launching `hydownloader-daemon` without any subscriptions added yet.
* The `mass-add-subscriptions` command can now set worker IDs
* Added a flag to `hydownloader-daemon` to not print every API request to the console
* Thanks to a user, the e621 tests has been rewritten/fixed
* Fixed crash in the importer when encountering non-string data in `additional_data`
* Updated yt-dlp to latest version

# 0.41.0 (2024-06-22)

* Updated gallery-dl and yt-dlp to latest versions
* Fixed newly created databases missing some columns in 0.40.0
* Updates to default config files (see the upgrade message for details)
* Fix some crashes introduced in the importer after a recent refactor

# 0.40.0 (2024-05-01)

* Updated gallery-dl and yt-dlp to latest versions
* Several updates have been made to the default configuration files (see the upgrade message for details)
* Added experimental support for multithreaded subscription checking, updated web UI to support this - READ THE DOCS ABOUT MULTITHREADED DOWNLOADING IF YOU WANT TO TRY THIS
* Columns are now reorderable in the web UI
* Added a 'last check status' column for subscriptions in the web UI
* Fixed uncaught exceptions in the URL normalizer causing crashes when encountering certain invalid "URL"s (mostly in the anchor-exporter)
* Updated URL patterns for x/twitter
* Added a HTML unescaping step
* Started work on auto-importing and import queue, refactored importer code

# 0.39.0 (2024-03-24)

* Removed the `convert-config` command from the importer.
  The new configuration format has been in use for a while and hopefully everyone has upgraded to it by now.
  If for some reason you still need to convert an old JSON configuration file,
  then you can download a copy of 0.38.0 and use that (the `convert-config` command doesn't touch
  the hydownloader database at all so it's safe to run it from an older version).
* Updated gallery-dl and yt-dlp to latest versions
* Possible fix in importer when trying to resume daemon after an import job

# 0.38.0 (2024-02-29)

* The `subscription-downloader-stats` command's output has more columns with subscription data.
* Fix: from now  on, subscriptions will ignore the `abort_after` value on initial check
* Fix: error on Python 3.9 in the importer due to use of 3.10-only feature
* Fix: the default importer rules had wrong pixiv namespaces (`pixiv id` should be used for user ID not work ID according to PTR convention)

# 0.37.0 (2024-02-18)

* Updated gallery-dl to 1.26.8
* Fix: the `recently-failing-subscriptions` command will now only count actually failing subscriptions (but an option was added to preserve the old behavior)
* The `--wait-for-daemon` flag in the importer now actually asks the daemon to pause, then waits until it pauses before importing,
  then asks the daemon to resume after importing is done (patch contribued by a user, thanks!)
* The importer can now set modification times (patch contributed by a user, thanks!):
  * Importer can now generate and set domain times
  * Importer will now set file modification time in Hydrus when not using path-based import,
    i.e. when Hydrus can't get modification time from the imported file because it doesn't see the filesystem metadata,
    like when importing over the network
  * Domain time rules added to the default import rules for danbooru, gelbooru and safebooru

# 0.36.0 (2024-01-22)

* Updated gallery-dl and yt-dlp to latest versions

# 0.35.0 (2024-01-07)

* Updated gallery-dl and yt-dlp to latest versions

# 0.34.0 (2023-12-24)

* Minor fixes in importer related to lolibooru support
* Fixed default safebooru importer rules to not error on empty (missing) source url
* Updated gallery-dl to 1.26.5

# 0.33.0 (2023-12-10)

* Updated Web UI to latest release, adding support for userCss option
* Fixed Web UI not working on some Windows systems due to files being served with the wrong filetype
* Fixes to the 0.32.0 upgrade process
* Updated gallery-dl to 1.26.4

# 0.32.0 (2023-12-04)

## Web UI

* Implemented experimental web UI (web version of the hydownloader-systray GUI).
  You can access it by navigating to `/webui` under your hydownloader daemon address.
  You can configure the web UI in hydownloader-config.json (see docs)

## Smaller fixes & additions

* The time to wait before rechecking an errored subscription is now configurable: `errored-sub-recheck-min-wait-seconds`
* The `url_info` API (and so Hydrus Companion) now recognizes existing raw subscriptions
* Bugfix: the root path returned by the API is now always an absolute path
* Various documentation updates
* Updated default import rules for imgur
* Reworked documentation structure (split up into more, shorter files)
* Added the `gallery-dl.do-not-use-cookiestxt` option which you can set in `hydownloader-config.json` to stop hydownloader from using the `cookies.txt` file in its database root path.
  This is needed if you want to set specific cookies in the gallery-dl configuration file,
  but then only those cookies will be used (using `cookies.txt` and cookies from the gallery-dl configuration at the same time is not possible)
* Implemented a new, more aggressive system to notify users about changes requiring manual updates to config files after a hydownloader version update.
  Now hydownloader won't start until any outstanding changes have been addressed
* `/api_version` now returns the hydownloader version
* Implemented archiving of subscriptions to hide old, no longer active subscriptions without actually deleting the entry from the database
  (works the same as archiving URLs)
* The configuration option `daemon.access_key` was renamed to `daemon.access-key` (but the old name will keep working too)
* Added `/rotate_daemon_log` API endpoint
* The `/get_subscriptions` API now also returns whether a subscriptions is due for a check
* Added `daemon.do-not-check-access-key` option to disable access key checks
* New API endpoint `/info` for getting the current/default configuration and root/data paths
* New API endpoint `/image_preview` to request image previews
* The `subscriptions_last_files` and `urls_last_files` API endpoints gained a new `limit` parameter (optional)
* The daemon now checks if its TCP port is actually available before doing anything at startup.
  Note that since after an application releases a TCP port it takes some time before it becomes available again,
  so quickly shutting down then starting the daemon again can result in it still seeing the port as in use
* Lowered the severity of 'missing configuration key' warnings when the missing option has a default value
  (happens if an older hydownloader-config.json is used that doesn't have some newly added option)
* The hydownloader version & database path is now printed to the log on startup

## New commands in hydownloader tools

* New command in hydownloader-tools: `subscription-downloader-stats` to generate summary statistics
  about the number (and optionally size) of new/already seen files, broken down by subscription ID and by downloader
* New command in hydownloader-tools: `recently-failing-subscriptions` to easily identify failing subscriptions
* Improved normalization of subscription keywords (e.g. twitter usernames are now case-insensitive and variants with any case will be recognized as the same).
  Related to this, `find-duplicate-subscriptions` was added to hydownloader-tools to help with finding any such already existing subscription duplicates

## Site support & 3rd party modules

* Fixes for Instagram support
* Added support for redgifs
* Added support for atfbooru
* Added support for misskey
* Fixed artstation test
* Fixed pixiv test
* Fixed redgifs URL regex
* Recognize more pixiv URL variations
* Updated kemono.party URLs to use new domain
* The default gallery-dl configuration was updated to also download novels from pixiv
* Updated gallery-dl, yt-dlp and Hydrus API module (the importer now requires Hydrus v514 or newer)

## Importer

* The import job configuration format is no longer JSON. A `convert-config` command was added to the importer to help with updating your configuration to the new format. See below
* The importer now uses service keys instead of names to identify tag repos, in accordance with recent Hydrus API changes.
  You can keep using tag repo names in your configuration but will have to provide the corresponding keys in the job config
* The importer now supports generating notes
* Minor fixes in how importer statistics are displayed after an import job finishes
* The importer now prints the name of the loaded configuration file into the log
* Added the `--early-hash-check` flag to hydownloader-importer,
  which can be used to write importer rules that do different things based on whether a file is already in Hydrus or not
* Updated importer rules to handle missing tags for misskey
* Updated importer rules for Twitter to avoid errors when parsing metadata files of direct image links (i.e. when instead of a tweet URL, only an image URL was sent to hydownloader)
* The importer now tries to check if the daemon is running and won't start import jobs in that case. Added the `--wait-for-daemon` and `--ignore-daemon` flags to control this behavior
* The importer now catches exceptions originating from Hydrus API calls and handles them gracefully
* Implemented a whitelist option in the importer for tags that end with a colon but are actually not misparsed
* Importer: `metadata` namespace is now rewritten to `meta` when using the `get_namespaces_tags` utility function
* Importer: some error messages should be more understandable now
* Importer: fixes in the default rules for rule34.xxx to correctly handle files which lack tag metadata fields split by namespace
* Importer: support for more variations of pixiv tag metadata in the default config
* Importer: do not record files as imported if the Hydrus API call to add the file fails, even if the 'abort on errors' flag is off
* Importer: print Hydrus' error message if an error is returned by the Hydrus API
* Importer: catch errors from missing files, and on Windows, also about too long paths (and print some info related to long path support)
* Updated default import rules for imgur
* Importer: only call the Hydrus API to add tags if there were actually tags generated for the file being imported
* Importer: added the `overridePathBasedLocation` option (see docs)
* Expanded default importer rules for twitter

### !!!IMPORTANT!!! About the new importer configuration format and the service name -> service key switch.

From this release, the primary way to define import jobs is no longer a JSON file.
Instead, import jobs are defined in Python code, which is loaded by the importer.
All features and configuration options remain the same, but this makes it much easier to write more complex
functions to generate tags/URLs/notes and variables can be used to avoid having to repeat certain parts in every single rule.
The readability of the configuration is also much improved (and it can have comments too).

On upgrade, a `hydownloader-import-jobs.py.NEW` file will be created in your hydownloader database folder (just like any other time when the default configuration files change).
You can check out this file to see the new configuration syntax.
If you were mostly using the default importer configuration, you can simply transfer your customizations and then switch to using this file (remove the .NEW extension so it won't get overwritten on next update!).
If you had a fully custom importer config, then you can use the `convert-config` command of the importer to generate a `.py`˙file from it.
The generated file will likely work as-is, but it might happen that some adjustments are needed.

Running the importer is still done exactly the same way as before (i.e. you have to specify which configuration file to load - or if you only specified the job name and used the default file, then it will prefer the new Python format if the `.py` file exists, but will use the old format if it doesn't).
The importer now also prints the name of the configuration file it's using to avoid any mixups between the old and new formats.
You can also still load the old format JSON files, but it is recommended to switch to the new format.

Another important change in this release is that the importer now uses tag service keys internally instead of names to identify tag repositories.
This change was required because of the recent Hydrus API move from using names to using keys.
You can keep using tag repo names in your import job configuration (much more readable), but will need to add the corresponding keys
to the config too, so the importer can translate names to keys before sending metadata to Hydrus.
The newly created default `.py.NEW` configuration file contains the name-to-key mappings for those tag repositories that Hydrus automatically creates ("local tags"/"my tags"/"downloader tags"),
but if you use an existing importer configuration file, or have any non-default tag repos, you will need to add the corresponding keys yourself.
This is done in the importer configuration, with the `serviceNamesToKeys` option. See the default configuration for an example on how to do it in the new Python format.
For the old JSON format, you need to add the `serviceNamesToKeys` key directly to the job configuration, which must be a JSON object where the keys are the service names, and the values
are the service keys (as strings).

# 0.31.0 (2023-01-11)

* Updated gallery-dl to 1.24.4
* Updated yt-dlp to 2023.1.6
* Fixed a configuration bug that broke artstation downloads with multiple files

# 0.30.0 (2022-12-18)

* Updated gallery-dl to 1.24.2

# 0.29.0 (2022-12-04)

* Updated gallery-dl to 1.24.1
* The 'version-metadata' key was added to the default gallery-dl-config.json (see the update log message for more info)
* Fixed tests for pixiv and furaffinity
* Bugfix: on Windows, CTRL_C_EVENT is used now instead of SIGINT, since the latter is not supported
* Documentation expanded with warnings about running import jobs and downloads at the same time and about tests failing with non-default filename formats

# 0.28.0 (2022-11-20)

* Updated gallery-dl to 1.24.0
* Updated yt-dlp to latest version
* Added an mtime postprocessor in the default gallery-dl configuration for artstation

# 0.27.0 (2022-10-29)

* Updated gallery-dl to 1.23.4

# 0.26.0 (2022-10-21)

* Updated gallery-dl to 1.23.3
* Updated yt-dlp to latest version
* Added support for aibooru.online (currently requires patched gallery-dl, as the relevant pull request is not merged yet into upstream gallery-dl)
* Added a command to hydownloader-tools to easily query & process danbooru banned artist data/URLs for mass scraping or analysis

# 0.25.0 (2022-10-01)

* Add the "downloaders" command to hydownloader-tools to list available downloaders (for subscriptions)
* Updated gallery-dl to 1.23.2

# 0.24.0 (2022-09-18)

* Updated gallery-dl to 1.23.1
* Updated yt-dlp and other dependencies to latest versions
* Fix a bug in the importer where it tried to remove some JSON files twice
* Log elapsed time in the importer
* Fix a bug where config files were not always opened with UTF-8 encoding on Windows
* Fix a bug in the hentaifoundry importer configuration that could lead to errors when trying to import "stories"

# 0.23.0 (2022-08-28)

* Updated gallery-dl to 1.23.0
* Updated yt-dlp
* Access key in the default configuration is now random instead of a fixed default
* hydownloader-tools gained a command to list subs with missed checks
* Implement remaining time estimation for subscription checks (hydownloader-systray will display this in the tray icon tooltip)
* Fix: check if database & data paths exists and are writeable and shut down if they aren't
* Fix: check for yt-dlp instead of youtube-dl in environment test
* Misc. fixes in tests
* Some work on reverse lookup mode (not yet usable)
* Misc. dependency updates

# 0.22.0 (2022-06-30)

* Updated gallery-dl to 1.22.3
* Updated yt-dlp and other dependencies
* New command line flags for better error handling and various other improvements in the importer

# 0.21.0 (2022-06-05)

* Updated gallery-dl to 1.22.1
* Instagram support

# 0.20.0 (2022-05-26)

* Updated dependencies (including gallery-dl update to 1.22.0)
* Added the `--subdir` command line argument to hydownloader-importer
* Added support for coomer.party, Furaffinity and e621
* Some more pixiv direct file URLs are recognized now
* Default importer rules: update rules for newgrounds URLs
* Default importer rules: fix gelbooru ID tags and URLs
* Default gallery-dl configuration: Pixiv artist profile backgrounds and avatars are also downloaded now (in addition to artworks)

# 0.19.0 (2022-04-28)

* Added command to hydownloader-tools for downloading Pixiv user profile data
* Updated gallery-dl (1.21.2)

# 0.18.0 (2022-04-09)

* Updated gallery-dl (1.21.1), yt-dlp and other dependencies
* Twitter age-gate is now circumvented (thanks to gallery-dl)
* Importer: catch a previously uncaught error in URL validity checking
* Filename format for kemono.party was updated to avoid excessively long filenames causing errors
* Support added for rule34.xxx
* The default values for multiple configuration files changed this release. Make sure to read the update message in the log, and update your configuration files accordingly.

# 0.17.0 (2022-03-23)

* Importer now generates a "gallerydl_file_url" value if possible, so that importer configurations do not need to be modified due to the changes in 0.16.0
* Importer: fixed error when sorting files by ctime or mtime
* Fixed error on startup on Windows


# 0.16.0 (2022-03-14)

* Added subscription check time statistics to the report
* Started implementation of reverse lookup features (can't be used yet)
* The value of the "url-metadata" gallery-dl option is now managed by hydownloader (for most users this shouldn't matter, see the upgrade message in the log for details)
* Upgraded gallery-dl to 1.21.0
* Upgraded yt-dlp to latest version
* Some other minor dependency upgrades

# 0.15.0 (2022-02-17)

* Added option to disable WAL journaling

# 0.14.0 (2022-02-16)

* Dependency updates (most importantly gallery-dl to 1.20.5)
* gelbooru: recognize sample URL variants
* Fixed some failing site tests
* Fixed ffmpeg hang due to SIGTTOU on Linux

# 0.13.0 (2022-02-06)

* Dependency updates (most importantly gallery-dl to 1.20.4)

# 0.12.0 (2022-01-28)

* Dependency updates (most importantly gallery-dl to 1.20.2)

# 0.11.0 (2022-01-09)

* Dependency updates (most importantly gallery-dl to 1.20.1)

# 0.10.0 (2021-12-18)

* Updated hydrus-api module
* Further fixes to handling forward vs. backslashes in filepaths, including updates to the default importer job configuration
* Updated dependencies (including youtube-dlp and youtube-dl)
* Log files are now UTF-8 encoded, even on Windows (the daemon log will be automatically rotated because of this when you upgrade your db)
* Do not error in some specific circumstances if the anchor db hasn't been created yet
* Fix failing pixiv test in hydownloader-tools
* Added Python version display to the environment test in hydownloader-tools
* Importer: do not try to associate URLs if there are no URLs for a file
* Importer: added some datetime conversion helper functions
* Configuration files are now checked on startup for correct syntax
* Interrupting the process with Ctrl+C should now actually wait for the currently running downloads to finish
* Added /subscription_data_to_url API endpoint
* Extended the tracking of missed subscription checks, now all instances of potentially missed files can be identified just from checking this list
* Documentation updates, including documenting the 'missed subscription checks' feature

# 0.9.0 (2021-11-28)

* Update gallery-dl to 1.19.3
* Update yt-dlp and some other dependencies to their latest versions

# 0.8.0 (2021-11-23)

* (experimental) Add tracking of missed subscription checks (either due to hydownloader being interrupted or severely exceeding check interval)
* Update gallery-dl to 1.19.2
* The importer now uses forward slashes (/) for filepaths on all platforms. This is a breaking change and the importer configuration might need to be updated
* Removed Nuitka from dev dependencies

# 0.7.0 (2021-10-25)

* Switch from youtube-dl to yt-dlp, as youtube-dl seems abandoned
* Update gallery-dl to 1.19.1

# 0.6.0 (2021-10-08)

* Fix: URLs were converted to lowercase in some log messages (downloading was unaffected)
* Fix: crash due wrong encoding on Windows when reading gallery-dl output (now hydownloader sets PYTHONIOENCODING=utf-8)
* Fix: URLs containing colons (fuck you twitter)
* Fix: gallery-dl configuration for twitter direct image links

# 0.5.0 (2021-10-02)

* New command added to hydownloader-tools: rotate-daemon-log
* New options added to the report feature to include/exclude archived URLs and paused subscriptions
* Changed how the order of due subs is determined, priority is the primary ordering value now
* The `get_subscription_checks` API endpoint was changed to allow retrieving check history for multiple subs at once
* Updated gallery-dl to 1.19.0 (configuration file update required, see log for details)

# 0.4.0 (2021-09-11)

* Updated gallery-dl to 1.18.4
* Updated some other dependencies to newest versions
* Added `"fallback": false` for twitter in the default gallery-dl user configuration
* Added ability to set default values for subscription/single URL properties in `hydownloader-config.json`