Compare commits

..

54 Commits

Author SHA1 Message Date
Kieran Eglin 75fb1a6ab9 Bumped version 2024-11-27 10:43:07 -08:00
Kieran 652fcccb4a [Enhancement] Add audio track language selection to Media Profile (#487)
* Moved quality options to their own module

* Added language and format selection to quality option builder

* [WIP] migrating tests

* Added audio_lang to media_profile table

* Renamed column; added format options and tests

* Adds UI for audio_track to the media profile form

* Adds a version string to in-app streams to help with cache busting
2024-11-27 10:39:29 -08:00
Kieran bfb27427ce [Bugfix] Ensure livestreams aren't downloaded until they're finished processing (#485)
* Added logic to ignore downloads that aren't in the right live state

* Added tests for get_downloadable_status method

* Added tests for media downloader module

* Added tests to download worker modeule
2024-11-26 11:56:33 -08:00
Kieran d9c48370df [Enhancement] Adds ability to enable/disable sources (#481)
* [Unrelated] updated module name for existing liveview module

* Updated toggle component and moved MP index table to a liveview

* [WIP] reverted MP index table; added source count to MP index

* Moved new live table to sources index

* Added 'enabled' boolean to sources

* Got 'enabled' logic working re: downloading pending media

* Updated sources context to do the right thing when a source is updated

* Docs and tests

* Updated slow indexing to maintain its old schedule if re-enabled

* Hooked up the enabled toggle to the sources page

* [Unrelated] added direct links to various tabs on the sources table

* More tests

* Removed unneeded guard in

* Removed outdated comment
2024-11-21 14:38:37 -08:00
Duong Nguyen 4c8c0461be [Enhancement] Add option to use existing Media Profile as template for new profile (#466)
* Add option to use existing Media Profile as template for new profile

* Forgot to commit the edit form too

* Reset deletion mark on Source controller

* Add test for new preload profile feature

* mix check
2024-11-20 10:25:53 -08:00
Kieran a02f71f304 [Enhancement] Add support for yt-dlp plugins + add lifecycle script event for app boot (#465)
* Added new script type to pre-job startup tasks

* Updated Dockerfile to create the needful directories

* added tests
2024-11-08 15:38:00 -08:00
Kieran 83c10b2b00 [Enhancement] Track the predicted final filepath for indexed media items (#461)
* Added ability to pass additional yt-dlp options to indexing step

* Added predicted_filename to media struct

* WIP added ability to predict filepath to source indexing

* renamed predicted_filepath

* Added the ability to predict filepath when fast indexing

* Add predicted_media_filepath to media items table

* Addressed TODOs
2024-11-08 09:42:59 -08:00
Kieran Eglin 8c0dd0bb6b Bumped version 2024-10-30 14:13:08 -07:00
Kieran Eglin 7a374f1162 Updated README 2024-10-30 14:12:29 -07:00
Kieran a5afcf168b Went back to using original_url over webpage_url to improve short detection (#447) 2024-10-30 14:09:28 -07:00
Kieran Eglin 2f83fe03d2 Updated default tzdata location to prevent needless pulls every time I push a new version 2024-10-30 10:18:03 -07:00
Kieran Eglin 3c10d78e78 Updated timezone logic to prevent boot crash 2024-10-30 10:14:04 -07:00
Kieran 8c0df65c0c Took the easy way out and removed playlist index explicitly from the update method (#442) 2024-10-28 10:26:31 -07:00
Kieran 47bb1cb98d Updated timezone logic to not prevent booting (#441) 2024-10-28 09:15:45 -07:00
Kieran Eglin 20547e60e4 Bump version 2024-10-25 11:42:16 -07:00
Kieran 0fd3106cb8 Removed root pw workaround (#437) 2024-10-25 11:17:16 -07:00
Kieran 108a141c65 Improve error messages when adding a non-valid source URL (#436) 2024-10-25 11:01:46 -07:00
Kieran 6a1b7b0160 [Enhancement] Misc. UI updates (#434)
* Format 'Scheduled At' in task table according to user's TZ

* Changed video preview layout to 1 column on all displays

* Made links in descriptions clickable

* Updated Copy JSON button to properly escape newline characters
2024-10-24 12:47:48 -07:00
Kieran 3c8d99196a [Bugfix] Fix off-by-1 error for retention date logic (#432)
* Added a sanity check test to the media context

* Improves logic for handing media item culling dates
2024-10-24 10:26:30 -07:00
Jonas Bergler cae86953a0 document that BASE_ROUTE_PATH requires the prefix to be stripped (#413) 2024-10-22 12:24:34 -07:00
FibreTTP f661747a0c Flip settings submenu chevron (#405) 2024-10-10 14:19:03 -07:00
Kieran Eglin 85feb67299 Bumped version 2024-10-02 14:44:47 -07:00
Kieran 4843d7d6c2 Updated tzinfo package (#402) 2024-10-02 14:38:14 -07:00
Kieran a0b9e49486 [Enhancement] Added ability to detect when files have been deleted (#399)
* Added function for updating a media item's filepaths on-disk

* Added placeholder action to source page

* Turned the file sync into a job and properly hooked it up to the controller
2024-09-26 12:39:31 -07:00
Kieran 0163e85e76 [Bugfix] prevent duplicate videos from being downloaded if the video's name changes (#396)
* Added methods for deleting outdated files

* Hooked up outdated file deletion to media download worker
2024-09-23 15:59:48 -07:00
Kieran Eglin 7fb6fa3af4 Bumped version 2024-09-12 14:39:07 -07:00
Kieran bb8788f03d [Dev] Version updates 2024-09-12 (#388)
* Updated some deps

* Updated phoenix
2024-09-12 13:04:49 -07:00
Kieran ed4434512b [Dev] Speed up CI (#385)
* Moved around copy steps

* no-op change

* Reverted the no-op change

* Testing if docker-compose volume is needed

* Nope, it's needed

* Testing alternate driver

* Add containerd

* Bump version

* Revert
2024-09-12 12:38:12 -07:00
Kieran f6344d43d2 [Bugfix] Disallow concurrent slow-indexing runs for the same source (#384)
* Updated deps in case that does anything

* Kills running slow-indexes when a new slow-index is enqueued

* Revert deps upgrade (saving that for its own PR)
2024-09-10 14:16:10 -07:00
Kieran e0745bdfbe [Enhancement] Allow configuration of media container format (#383)
* Adds media container to media_profiles and updates option builder

* Added media container to media profile form
2024-09-10 11:44:57 -07:00
Kieran Eglin a0a02cb4be Updated docker-build-push version in docker release file as well 2024-09-10 11:42:10 -07:00
Kieran Eglin 7866f7258a Updated docker-build-push version 2024-09-10 11:36:54 -07:00
Kieran 5cd82f9dd5 Updated the aspect ratio threshold for short detection (#382) 2024-09-10 08:49:30 -07:00
Kieran Eglin c85e9cb5f1 Retry with a non-yanked version of ffmpeg I guess? 2024-08-28 09:58:53 -07:00
Kieran Eglin 4837dd44c9 Bumped version 2024-08-28 08:47:13 -07:00
Kieran Eglin e3f42c28d0 Opps - reversed the IPv6 logic 2024-08-27 20:25:53 -07:00
Kieran 4a11d47692 Locked ffmpeg to last known-good version (#373) 2024-08-27 20:20:53 -07:00
Kieran 612c760755 [Enhancement] Allow IPv6 (#372)
* Added option to enable IPv6 in runtime config

* Updated README
2024-08-27 20:08:53 -07:00
Kieran 14b8ecbe44 [Enhancement] Show error messages in-app for easier triage (#365)
* Added an improved error message screen with actionable details

* Added a basic 404 page

* fixed tests
2024-08-23 13:33:27 -07:00
Kieran a6c61ccd0d [Bugfix] Improve livestream detection (#362)
* Livestream status is now based off live_status instead of was_live

* Updated tests
2024-08-20 09:37:42 -07:00
Kieran Eglin 80209240d2 Updated logger to add dates in logfiles rather than just console output 2024-08-20 08:58:30 -07:00
Kieran Eglin 1d5b62ccf9 Bumped version 2024-08-14 14:48:58 -07:00
Kieran af8235c6b3 [Enhancement] Filter media based on min and/or max duration (#356)
* Added duration limits to source model

* Added duration limits to source form

* Added validation for min/max amounts

* Added duration checks to pending query

* Moved min/max filters up in source form

* Removed debugger
2024-08-14 14:20:50 -07:00
Kieran 8e9f02c807 [Enhancement] Support specifying count of yt-dlp workers per-queue (#355)
* Updated certain Oban queues to have a configurable limit

* Updated README
2024-08-14 12:37:24 -07:00
Kieran fd20ac5d84 [Enhancement] Use cookies on a per-source basis (#354)
* Added 'use_cookies' column to source

* Added cookies to the source form

* Updated every command to optionally include cookies

* LOTS of tests

* Made YT cookies off by default
2024-08-14 12:00:18 -07:00
Kieran 23492961d5 [Dev] Adds .devcontainer manifest to help promote contributions (#342)
* Added a devcontainer file

* linting

* Added link to README
2024-08-07 15:38:06 -07:00
Kieran Eglin 2326775d78 Bump for new yt-dlp version 2024-08-06 14:17:14 -07:00
Kieran Eglin e0a84a521b Bumped version 2024-07-30 08:23:03 -07:00
Kieran Eglin d217ad5744 [Dev] Added ZSH to dev dockerfile 2024-07-30 08:20:55 -07:00
Kieran d5ae41cdab [Performance] Asyncronously delete sources and media profiles (#277)
* [WIP] started on source deletion

* Removed unneeded test blocks

* Added marked_for_deletion_at to sources and media profiles

* Hooked up async deletion to media profiles as well
2024-07-22 12:26:22 -07:00
Kieran 7a01db05dd [Enhancement] Ignore media based on user scripts (#330)
* Integrated pre-download callback for media items

* Refactored existing tests

* Docs, etc
2024-07-22 10:47:49 -07:00
Kieran d392fc3818 [Enhancement] Media from before a source's cutoff date is now automatically deleted (#329)
* Changed the purpose of the 'culled_at' flag. renamed some methods

* implemented new culling behaviour for source cutoff dates

* Removed culled_at flag if a media item gets redownloaded

* Updated source form

* Removed unused method
2024-07-19 16:39:30 -07:00
Kieran Eglin 47596d5f72 Added a script to help with version bumping 2024-07-18 15:59:31 -07:00
Kieran Eglin dbc25e5730 changed version number because I'm a coward 2024-07-18 11:13:03 -07:00
121 changed files with 3334 additions and 908 deletions
+12
View File
@@ -0,0 +1,12 @@
{
"name": "Pinchflat Dev",
"dockerComposeFile": "../docker-compose.yml",
"service": "phx",
"workspaceFolder": "/app",
"shutdownAction": "stopCompose",
"customizations": {
"vscode": {
"extensions": ["phoenixframework.phoenix", "JakeBecker.elixir-ls", "esbenp.prettier-vscode"]
}
}
}
+1 -1
View File
@@ -72,7 +72,7 @@ jobs:
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build and Push
uses: docker/build-push-action@v5
uses: docker/build-push-action@v6
with:
context: .
file: ./docker/selfhosted.Dockerfile
+1 -1
View File
@@ -29,7 +29,7 @@ jobs:
uses: docker/setup-buildx-action@v3
- name: Build docker image
uses: docker/build-push-action@v5
uses: docker/build-push-action@v6
with:
context: .
file: ./docker/dev.Dockerfile
+8
View File
@@ -23,3 +23,11 @@ alias Pinchflat.Metadata.MetadataFileHelpers
alias Pinchflat.SlowIndexing.FileFollowerServer
Pinchflat.Release.check_file_permissions()
defmodule IexHelpers do
def restart do
:init.restart()
end
end
import IexHelpers
+1
View File
@@ -0,0 +1 @@
assets/vendor/
+17 -17
View File
@@ -17,6 +17,7 @@
[![](https://img.shields.io/github/license/kieraneglin/pinchflat?style=for-the-badge&color=ee512b)](LICENSE)
[![](https://img.shields.io/github/v/release/kieraneglin/pinchflat?style=for-the-badge)](https://github.com/kieraneglin/pinchflat/releases)
[![](https://img.shields.io/github/actions/workflow/status/kieraneglin/pinchflat/lint_and_test.yml?style=for-the-badge)](#)
[![](https://img.shields.io/static/v1?label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode&style=for-the-badge)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/kieraneglin/pinchflat)
</div>
@@ -52,7 +53,7 @@ If it doesn't work for your use case, please make a feature request! You can als
- Self-contained - just one Docker container with no external dependencies
- Powerful naming system so content is stored where and how you want it
- Easy-to-use web interface with presets to get you started right away
- First-class support for media center apps like Plex, Jellyfin, and Kodi
- First-class support for media center apps like Plex, Jellyfin, and Kodi ([docs](https://github.com/kieraneglin/pinchflat/wiki/Frequently-Asked-Questions#how-do-i-get-media-into-plexjellyfinkodi))
- Supports serving RSS feeds to your favourite podcast app ([docs](https://github.com/kieraneglin/pinchflat/wiki/Podcast-RSS-Feeds))
- Automatically downloads new content from channels and playlists
- Uses a novel approach to download new content more quickly than other apps
@@ -62,7 +63,7 @@ If it doesn't work for your use case, please make a feature request! You can als
- Allows automatically redownloading new media after a set period
- This can help improve the download quality of new content or improve SponsorBlock tags
- Optionally automatically delete old content ([docs](https://github.com/kieraneglin/pinchflat/wiki/Automatically-Delete-Media))
- Advanced options like setting cutoff dates and filtering by title
- Advanced options like setting cutoff dates and filtering by title ([docs](https://github.com/kieraneglin/pinchflat/wiki/Frequently-Asked-Questions#i-only-want-certain-videos-from-a-source---how-can-i-only-download-those))
- Reliable hands-off operation
- Can pass cookies to YouTube to download your private playlists ([docs](https://github.com/kieraneglin/pinchflat/wiki/YouTube-Cookies))
- Sponsorblock integration
@@ -130,9 +131,6 @@ You _must_ ensure the host directories you've mounted are writable by the user r
> [!IMPORTANT]
> It's not recommended to run the container as root. Doing so can create permission issues if other apps need to work with the downloaded media.
> [!TIP]
> If you need to run any command as root, you can run `su` from the container's shell as there is no password set for the root user.
### ADVANCED: Storing Pinchflat config directory on a network share
As pointed out in [#137](https://github.com/kieraneglin/pinchflat/issues/137), SQLite doesn't like being run in WAL mode on network shares. If you're running Pinchflat on a network share, you can disable WAL mode by setting the `JOURNAL_MODE` environment variable to `delete`. This will make Pinchflat run in rollback journal mode which is less performant but should work on network shares.
@@ -144,16 +142,18 @@ If you change this setting and it works well for you, please leave a comment on
### Environment variables
| Name | Required? | Default | Notes |
| --------------------- | --------- | ------------------------- | ---------------------------------------------------------------------------------------------- |
| TZ | No | `UTC` | Must follow IANA TZ format |
| LOG_LEVEL | No | `debug` | Can be set to `info` |
| BASIC_AUTH_USERNAME | No | | See [authentication docs](https://github.com/kieraneglin/pinchflat/wiki/Username-and-Password) |
| BASIC_AUTH_PASSWORD | No | | See [authentication docs](https://github.com/kieraneglin/pinchflat/wiki/Username-and-Password) |
| EXPOSE_FEED_ENDPOINTS | No | | See [RSS feed docs](https://github.com/kieraneglin/pinchflat/wiki/Podcast-RSS-Feeds) |
| JOURNAL_MODE | No | `wal` | Set to `delete` if your config directory is stored on a network share (not recommended) |
| TZ_DATA_DIR | No | `/etc/elixir_tzdata_data` | The container path where the timezone database is stored |
| BASE_ROUTE_PATH | No | `/` | The base path for route generation. Useful when running behind certain reverse proxies |
| Name | Required? | Default | Notes |
| --------------------------- | --------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `TZ` | No | `UTC` | Must follow IANA TZ format |
| `LOG_LEVEL` | No | `debug` | Can be set to `info` but `debug` is strongly recommended |
| `BASIC_AUTH_USERNAME` | No | | See [authentication docs](https://github.com/kieraneglin/pinchflat/wiki/Username-and-Password) |
| `BASIC_AUTH_PASSWORD` | No | | See [authentication docs](https://github.com/kieraneglin/pinchflat/wiki/Username-and-Password) |
| `EXPOSE_FEED_ENDPOINTS` | No | `false` | See [RSS feed docs](https://github.com/kieraneglin/pinchflat/wiki/Podcast-RSS-Feeds) |
| `ENABLE_IPV6` | No | `false` | Setting to _any_ non-blank value will enable IPv6 |
| `JOURNAL_MODE` | No | `wal` | Set to `delete` if your config directory is stored on a network share (not recommended) |
| `TZ_DATA_DIR` | No | `/etc/elixir_tzdata_data` | The container path where the timezone database is stored |
| `BASE_ROUTE_PATH` | No | `/` | The base path for route generation. Useful when running behind certain reverse proxies, but prefix must be stripped. |
| `YT_DLP_WORKER_CONCURRENCY` | No | `2` | The number of concurrent workers that use `yt-dlp` _per queue_. Set to 1 if you're getting IP limited, otherwise don't touch it |
## EFF donations
@@ -161,9 +161,9 @@ Prior to 2024-05-10, a portion of all donations were given to the [Electronic Fr
The EFF defends your online liberties and [backed](https://github.com/github/dmca/blob/9a85e0f021f7967af80e186b890776a50443f06c/2020/11/2020-11-16-RIAA-reversal-effletter.pdf) `youtube-dl` when Google took them down.
## Pre-release disclaimer
## Stability disclaimer
This is pre-release software and anything can break at any time. I make not guarantees about the stability of this software, forward-compatibility of updates, or integrity (both related to and independent of Pinchflat). Essentially, use at your own risk and expect there will be rough edges for now.
This software is in active development and anything can break at any time. I make no guarantees about the stability of this software, forward-compatibility of updates, or integrity (both related to and independent of Pinchflat).
## License
+7
View File
@@ -35,3 +35,10 @@ window.markVersionAsSeen = (versionString) => {
window.isVersionSeen = (versionString) => {
return localStorage.getItem('seenVersion') === versionString
}
window.dispatchFor = (elementOrId, eventName, detail = {}) => {
const element =
typeof elementOrId === 'string' ? document.getElementById(elementOrId) : elementOrId
element.dispatchEvent(new CustomEvent(eventName, { detail }))
}
+24 -1
View File
@@ -39,7 +39,7 @@ let liveSocket = new LiveSocket(document.body.dataset.socketPath, Socket, {
}
},
hooks: {
supressEnterSubmission: {
'supress-enter-submission': {
mounted() {
this.el.addEventListener('keypress', (event) => {
if (event.key === 'Enter') {
@@ -47,6 +47,29 @@ let liveSocket = new LiveSocket(document.body.dataset.socketPath, Socket, {
}
})
}
},
'formless-input': {
mounted() {
const subscribedEvents = this.el.dataset.subscribe.split(' ')
const eventName = this.el.dataset.eventName || ''
const identifier = this.el.dataset.identifier || ''
subscribedEvents.forEach((domEvent) => {
this.el.addEventListener(domEvent, () => {
// This ensures that the event is pushed to the server after the input value has been updated
// so that the server has the most up-to-date value
setTimeout(() => {
this.pushEvent('formless-input', {
value: this.el.value,
id: identifier,
event: eventName,
dom_id: this.el.id,
dom_event: domEvent
})
}, 0)
})
})
}
}
}
})
+3 -12
View File
@@ -41,7 +41,8 @@ config :pinchflat, PinchflatWeb.Endpoint,
adapter: Phoenix.Endpoint.Cowboy2Adapter,
render_errors: [
formats: [html: PinchflatWeb.ErrorHTML, json: PinchflatWeb.ErrorJSON],
layout: false
root_layout: {PinchflatWeb.Layouts, :root},
layout: {PinchflatWeb.Layouts, :app}
],
pubsub_server: Pinchflat.PubSub,
live_view: [signing_salt: "/t5878kO"]
@@ -57,16 +58,6 @@ config :pinchflat, Oban,
{"0 1 * * *", Pinchflat.Downloading.MediaRetentionWorker},
{"0 2 * * *", Pinchflat.Downloading.MediaQualityUpgradeWorker}
]}
],
# TODO: consider making this an env var or something?
queues: [
default: 10,
fast_indexing: 6,
media_indexing: 2,
media_collection_indexing: 2,
media_fetching: 2,
local_metadata: 8,
remote_metadata: 4
]
# Configures the mailer
@@ -101,7 +92,7 @@ config :tailwind,
]
# Configures Elixir's Logger
config :logger, :console,
config :logger, :default_formatter,
format: "$date $time $metadata[$level] | $message\n",
metadata: [:request_id]
+1 -1
View File
@@ -67,7 +67,7 @@ config :pinchflat, PinchflatWeb.Endpoint,
config :pinchflat, dev_routes: true
# Do not include metadata nor timestamps in development logs
config :logger, :console, format: "[$level] $message\n"
config :logger, :default_formatter, format: "[$level] $message\n"
# Set a higher stacktrace during development. Avoid configuring such
# in production as building large stacktraces may be expensive.
+19 -3
View File
@@ -40,6 +40,20 @@ config :pinchflat, Pinchflat.Repo,
Path.join([:code.priv_dir(:pinchflat), "repo", "extensions", "sqlean-linux-#{system_arch}", "sqlean"])
]
# Some users may want to increase the number of workers that use yt-dlp to improve speeds
# Others may want to decrease the number of these workers to lessen the chance of an IP ban
{yt_dlp_worker_count, _} = Integer.parse(System.get_env("YT_DLP_WORKER_CONCURRENCY", "2"))
config :pinchflat, Oban,
queues: [
default: 10,
fast_indexing: 6,
media_collection_indexing: yt_dlp_worker_count,
media_fetching: yt_dlp_worker_count,
remote_metadata: yt_dlp_worker_count,
local_data: 8
]
if config_env() == :prod do
config_path = "/config"
db_path = System.get_env("DATABASE_PATH", Path.join([config_path, "db", "pinchflat.db"]))
@@ -52,6 +66,7 @@ if config_env() == :prod do
journal_mode = String.to_existing_atom(System.get_env("JOURNAL_MODE", "wal"))
# For running PF in a subdirectory via a reverse proxy
base_route_path = System.get_env("BASE_ROUTE_PATH", "/")
enable_ipv6 = String.length(System.get_env("ENABLE_IPV6", "")) > 0
config :logger, level: String.to_existing_atom(System.get_env("LOG_LEVEL", "debug"))
@@ -64,11 +79,12 @@ if config_env() == :prod do
tmpfile_directory: Path.join([System.tmp_dir!(), "pinchflat", "data"]),
dns_cluster_query: System.get_env("DNS_CLUSTER_QUERY"),
expose_feed_endpoints: expose_feed_endpoints,
timezone: System.get_env("TIMEZONE") || System.get_env("TZ") || "UTC",
# This is configured in application.ex
timezone: "UTC",
log_path: log_path,
base_route_path: base_route_path
config :tzdata, :data_dir, System.get_env("TZ_DATA_DIR", "/etc/elixir_tzdata_data")
config :tzdata, :data_dir, System.get_env("TZ_DATA_DIR", "/config/extras/elixir_tz_data")
config :pinchflat, Pinchflat.Repo,
database: db_path,
@@ -106,7 +122,7 @@ if config_env() == :prod do
# Set it to {0, 0, 0, 0, 0, 0, 0, 1} for local network only access.
# See the documentation on https://hexdocs.pm/plug_cowboy/Plug.Cowboy.html
# for details about using IPv6 vs IPv4 and loopback vs public addresses.
ip: {0, 0, 0, 0},
ip: if(enable_ipv6, do: {0, 0, 0, 0, 0, 0, 0, 0}, else: {0, 0, 0, 0}),
port: String.to_integer(System.get_env("PORT") || "4000")
],
url: [path: base_route_path],
-2
View File
@@ -10,5 +10,3 @@ services:
command: bash -c "chmod +x docker/docker-run.dev.sh && docker/docker-run.dev.sh"
stdin_open: true
tty: true
env_file:
- .env
+10 -5
View File
@@ -12,7 +12,7 @@ RUN echo "Building for ${TARGETPLATFORM:?}"
RUN apt-get update -qq && \
apt-get install -y inotify-tools curl git openssh-client jq \
python3 python3-setuptools python3-wheel python3-dev pipx \
python3-mutagen locales procps build-essential graphviz
python3-mutagen locales procps build-essential graphviz zsh
# Install ffmpeg
RUN export FFMPEG_DOWNLOAD=$(case ${TARGETPLATFORM:-linux/amd64} in \
@@ -38,7 +38,10 @@ RUN curl -sL https://deb.nodesource.com/setup_20.x -o nodesource_setup.sh && \
# Install Apprise
export PIPX_HOME=/opt/pipx && \
export PIPX_BIN_DIR=/usr/local/bin && \
pipx install apprise
pipx install apprise && \
# Set up ZSH tools
chsh -s $(which zsh) && \
sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"
# Set the locale
RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen
@@ -46,14 +49,16 @@ ENV LANG=en_US.UTF-8
ENV LANGUAGE=en_US:en
ENV LC_ALL=en_US.UTF-8
# Create app directory and copy the Elixir projects into it.
WORKDIR /app
COPY . ./
COPY mix.exs mix.lock ./
# Install Elixir deps
# RUN mix archive.install github hexpm/hex branch latest
# NOTE: this has to be before the bulk copy to ensure that deps are cached
RUN MIX_ENV=dev mix deps.get && MIX_ENV=dev mix deps.compile
RUN MIX_ENV=test mix deps.get && MIX_ENV=test mix deps.compile
COPY . ./
# Gives us iex shell history
ENV ERL_AFLAGS="-kernel shell_history enabled"
+6 -15
View File
@@ -27,10 +27,10 @@ RUN apt-get update -y && \
# Hex and Rebar
mix local.hex --force && \
mix local.rebar --force && \
# FFmpeg
# FFmpeg (latest build that doesn't cause an illegal instruction error for some users - see #347)
export FFMPEG_DOWNLOAD=$(case ${TARGETPLATFORM:-linux/amd64} in \
"linux/amd64") echo "https://github.com/yt-dlp/FFmpeg-Builds/releases/download/latest/ffmpeg-master-latest-linux64-gpl.tar.xz" ;; \
"linux/arm64") echo "https://github.com/yt-dlp/FFmpeg-Builds/releases/download/latest/ffmpeg-master-latest-linuxarm64-gpl.tar.xz" ;; \
"linux/amd64") echo "https://github.com/yt-dlp/FFmpeg-Builds/releases/download/autobuild-2024-07-30-14-10/ffmpeg-N-116468-g0e09f6d690-linux64-gpl.tar.xz" ;; \
"linux/arm64") echo "https://github.com/yt-dlp/FFmpeg-Builds/releases/download/autobuild-2024-07-30-14-10/ffmpeg-N-116468-g0e09f6d690-linuxarm64-gpl.tar.xz" ;; \
*) echo "" ;; esac) && \
curl -L ${FFMPEG_DOWNLOAD} --output /tmp/ffmpeg.tar.xz && \
tar -xf /tmp/ffmpeg.tar.xz --strip-components=2 --no-anchored -C /usr/local/bin/ "ffmpeg" && \
@@ -88,6 +88,7 @@ RUN apt-get update -y && \
ca-certificates \
python3-mutagen \
curl \
zip \
openssh-client \
nano \
python3 \
@@ -116,7 +117,8 @@ ENV LC_ALL en_US.UTF-8
WORKDIR "/app"
# Set up data volumes
RUN mkdir /config /downloads /etc/elixir_tzdata_data && chmod ugo+rw /etc/elixir_tzdata_data
RUN mkdir -p /config /downloads /etc/elixir_tzdata_data /etc/yt-dlp/plugins && \
chmod ugo+rw /etc/elixir_tzdata_data /etc/yt-dlp /etc/yt-dlp/plugins
# set runner ENV
ENV MIX_ENV="prod"
@@ -127,17 +129,6 @@ EXPOSE ${PORT}
# Only copy the final release from the build stage
COPY --from=builder /app/_build/${MIX_ENV}/rel/pinchflat ./
# NEVER do this if you're running in an environment where you don't trust the user
# (ie: most environments). This is only acceptable in a self-hosted environment.
# The user could just run the whole container as root and bypass this anyway so
# it's not a huge deal.
# This removes the root password to allow users to assume root if needed. This is
# preferrable to running the whole container as root so that the files/directories
# created by the app aren't owned by root and are therefore easier for other users
# and processes to interact with. If you want to just run the whole container as
# root, use --user 0:0 or something.
RUN passwd -d root
HEALTHCHECK --interval=120s --start-period=10s \
CMD curl --fail http://localhost:${PORT}/healthcheck || exit 1
+19
View File
@@ -4,9 +4,12 @@ defmodule Pinchflat.Application do
@moduledoc false
use Application
require Logger
@impl true
def start(_type, _args) do
check_and_update_timezone()
children = [
PinchflatWeb.Telemetry,
Pinchflat.Repo,
@@ -47,4 +50,20 @@ defmodule Pinchflat.Application do
:ok = Oban.Telemetry.attach_default_logger()
:telemetry.attach_many("job-telemetry-broadcast", events, &PinchflatWeb.Telemetry.job_state_change_broadcast/4, [])
end
# This has to be here (rather than runtime.exs) since the `tzdata` application
# has to be started before we can check the timezone
defp check_and_update_timezone do
attempted_timezone = System.get_env("TIMEZONE") || System.get_env("TZ") || "UTC"
valid_timezone =
if Tzdata.zone_exists?(attempted_timezone) do
attempted_timezone
else
Logger.warning("Invalid timezone #{attempted_timezone}, defaulting to UTC")
"UTC"
end
Application.put_env(:pinchflat, :timezone, valid_timezone)
end
end
@@ -16,6 +16,8 @@ defmodule Pinchflat.Boot.PreJobStartupTasks do
alias Pinchflat.Settings
alias Pinchflat.Utils.FilesystemUtils
alias Pinchflat.Lifecycle.UserScripts.CommandRunner, as: UserScriptRunner
def start_link(opts \\ []) do
GenServer.start_link(__MODULE__, %{}, opts)
end
@@ -36,6 +38,7 @@ defmodule Pinchflat.Boot.PreJobStartupTasks do
create_blank_yt_dlp_files()
create_blank_user_script_file()
apply_default_settings()
run_app_init_script()
{:ok, state}
end
@@ -95,6 +98,12 @@ defmodule Pinchflat.Boot.PreJobStartupTasks do
Settings.set(apprise_version: apprise_version)
end
defp run_app_init_script do
runner = Application.get_env(:pinchflat, :user_script_runner, UserScriptRunner)
runner.run(:app_init, %{})
end
defp yt_dlp_runner do
Application.get_env(:pinchflat, :yt_dlp_runner)
end
@@ -4,10 +4,10 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilder do
"""
alias Pinchflat.Sources
alias Pinchflat.Settings
alias Pinchflat.Sources.Source
alias Pinchflat.Media.MediaItem
alias Pinchflat.Downloading.OutputPathBuilder
alias Pinchflat.Downloading.QualityOptionBuilder
alias Pinchflat.Utils.FilesystemUtils, as: FSUtils
@@ -34,21 +34,38 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilder do
@doc """
Builds the output path for yt-dlp to download media based on the given source's
media profile. Uses the source's override output path template if it exists.
or media_item's media profile. Uses the source's override output path template if it exists.
Accepts a %MediaItem{} or %Source{} struct. If a %Source{} struct is passed, it
will use a default %MediaItem{} struct with the given source.
Returns binary()
"""
def build_output_path_for(%Source{} = source_with_preloads) do
build_output_path_for(%MediaItem{source: source_with_preloads})
end
def build_output_path_for(%MediaItem{} = media_item_with_preloads) do
output_path_template = Sources.output_path_template(media_item_with_preloads.source)
build_output_path(output_path_template, media_item_with_preloads)
end
def build_output_path_for(%Source{} = source_with_preloads) do
build_output_path_for(%MediaItem{source: source_with_preloads})
@doc """
Builds the quality options for yt-dlp to download media based on the given source's
or media_item's media profile. Useful for helping predict final filepath of downloaded
media.
returns [Keyword.t()]
"""
def build_quality_options_for(%Source{} = source_with_preloads) do
build_quality_options_for(%MediaItem{source: source_with_preloads})
end
def build_quality_options_for(%MediaItem{} = media_item_with_preloads) do
media_profile = media_item_with_preloads.source.media_profile
quality_options(media_profile)
end
defp default_options(override_opts) do
@@ -125,26 +142,7 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilder do
end
defp quality_options(media_profile) do
vcodec = Settings.get!(:video_codec_preference)
acodec = Settings.get!(:audio_codec_preference)
case media_profile.preferred_resolution do
# Also be aware that :audio disabled all embedding options for subtitles
:audio ->
[:extract_audio, format_sort: "+acodec:#{acodec}"]
resolution_atom ->
{resolution_string, _} =
resolution_atom
|> Atom.to_string()
|> Integer.parse()
[
# Since Plex doesn't support reading metadata from MKV
remux_video: "mp4",
format_sort: "res:#{resolution_string},+codec:#{vcodec}:#{acodec}"
]
end
QualityOptionBuilder.build(media_profile)
end
defp sponsorblock_options(media_profile) do
@@ -91,8 +91,7 @@ defmodule Pinchflat.Downloading.DownloadingHelpers do
[m, s, mp],
^MediaQuery.for_source(source) and
^MediaQuery.downloaded() and
not (^MediaQuery.download_prevented()) and
not (^MediaQuery.culled())
not (^MediaQuery.download_prevented())
)
)
|> Repo.all()
@@ -12,6 +12,7 @@ defmodule Pinchflat.Downloading.MediaDownloadWorker do
alias Pinchflat.Tasks
alias Pinchflat.Repo
alias Pinchflat.Media
alias Pinchflat.Media.FileSyncing
alias Pinchflat.Downloading.MediaDownloader
alias Pinchflat.Lifecycle.UserScripts.CommandRunner, as: UserScriptRunner
@@ -39,17 +40,14 @@ defmodule Pinchflat.Downloading.MediaDownloadWorker do
- `quality_upgrade?`: re-downloads media, including the video. Does not force download
if the source is set to not download media
Returns :ok | {:ok, %MediaItem{}} | {:error, any, ...any}
Returns :ok | {:error, any, ...any}
"""
@impl Oban.Worker
def perform(%Oban.Job{args: %{"id" => media_item_id} = args}) do
should_force = Map.get(args, "force", false)
is_quality_upgrade = Map.get(args, "quality_upgrade?", false)
media_item =
media_item_id
|> Media.get_media_item!()
|> Repo.preload(:source)
media_item = fetch_and_run_prevent_download_user_script(media_item_id)
# If the source or media item is set to not download media, perform a no-op unless forced
if (media_item.source.download_media && !media_item.prevent_download) || should_force do
@@ -62,6 +60,20 @@ defmodule Pinchflat.Downloading.MediaDownloadWorker do
Ecto.StaleEntryError -> Logger.info("#{__MODULE__} discarded: media item #{media_item_id} stale")
end
# If a user script exists and, when run, returns a non-zero exit code, prevent this and all future downloads
# of the media item.
defp fetch_and_run_prevent_download_user_script(media_item_id) do
media_item = Media.get_media_item!(media_item_id)
{:ok, media_item} =
case run_user_script(:media_pre_download, media_item) do
{:ok, _, exit_code} when exit_code != 0 -> Media.update_media_item(media_item, %{prevent_download: true})
_ -> {:ok, media_item}
end
Repo.preload(media_item, :source)
end
defp download_media_and_schedule_jobs(media_item, is_quality_upgrade, should_force) do
overwrite_behaviour = if should_force || is_quality_upgrade, do: :force_overwrites, else: :no_force_overwrites
override_opts = [overwrite_behaviour: overwrite_behaviour]
@@ -74,13 +86,17 @@ defmodule Pinchflat.Downloading.MediaDownloadWorker do
media_redownloaded_at: get_redownloaded_at(is_quality_upgrade)
})
:ok = run_user_script(updated_media_item)
:ok = FileSyncing.delete_outdated_files(media_item, updated_media_item)
run_user_script(:media_downloaded, updated_media_item)
{:ok, updated_media_item}
:ok
{:recovered, _} ->
{:error, :retry}
{:error, :unsuitable_for_download} ->
{:ok, :non_retry}
{:error, message} ->
action_on_error(message)
end
@@ -112,9 +128,9 @@ defmodule Pinchflat.Downloading.MediaDownloadWorker do
# NOTE: I like this pattern of using the default value so that I don't have to
# define it in config.exs (and friends). Consider using this elsewhere.
defp run_user_script(media_item) do
defp run_user_script(event, media_item) do
runner = Application.get_env(:pinchflat, :user_script_runner, UserScriptRunner)
runner.run(:media_downloaded, media_item)
runner.run(event, media_item)
end
end
+14 -1
View File
@@ -37,6 +37,13 @@ defmodule Pinchflat.Downloading.MediaDownloader do
{:ok, parsed_json} ->
update_media_item_from_parsed_json(media_with_preloads, parsed_json)
{:error, :unsuitable_for_download} ->
Logger.warning(
"Media item ##{media_with_preloads.id} isn't suitable for download yet. May be an active or processing live stream"
)
{:error, :unsuitable_for_download}
{:error, message, _exit_code} ->
Logger.error("yt-dlp download error for media item ##{media_with_preloads.id}: #{inspect(message)}")
@@ -79,6 +86,7 @@ defmodule Pinchflat.Downloading.MediaDownloader do
|> MetadataParser.parse_for_media_item()
|> Map.merge(%{
media_downloaded_at: DateTime.utc_now(),
culled_at: nil,
nfo_filepath: determine_nfo_filepath(media_with_preloads, parsed_json),
metadata: %{
# IDEA: might be worth kicking off a job for this since thumbnail fetching
@@ -105,8 +113,13 @@ defmodule Pinchflat.Downloading.MediaDownloader do
defp download_with_options(url, item_with_preloads, output_filepath, override_opts) do
{:ok, options} = DownloadOptionBuilder.build(item_with_preloads, override_opts)
runner_opts = [output_filepath: output_filepath, use_cookies: item_with_preloads.source.use_cookies]
YtDlpMedia.download(url, options, output_filepath: output_filepath)
case YtDlpMedia.get_downloadable_status(url) do
{:ok, :downloadable} -> YtDlpMedia.download(url, options, runner_opts)
{:ok, :ignorable} -> {:error, :unsuitable_for_download}
err -> err
end
end
defp recoverable_errors do
@@ -23,10 +23,10 @@ defmodule Pinchflat.Downloading.MediaQualityUpgradeWorker do
"""
@impl Oban.Worker
def perform(%Oban.Job{}) do
redownloadable_media = Media.list_redownloadable_media_items()
Logger.info("Redownloading #{length(redownloadable_media)} media items")
upgradable_media = Media.list_upgradeable_media_items()
Logger.info("Redownloading #{length(upgradable_media)} media items")
Enum.each(redownloadable_media, fn media_item ->
Enum.each(upgradable_media, fn media_item ->
MediaDownloadWorker.kickoff_with_task(media_item, %{quality_upgrade?: true})
end)
end
@@ -2,12 +2,15 @@ defmodule Pinchflat.Downloading.MediaRetentionWorker do
@moduledoc false
use Oban.Worker,
queue: :local_metadata,
queue: :local_data,
unique: [period: :infinity, states: [:available, :scheduled, :retryable, :executing]],
tags: ["media_item", "local_metadata"]
tags: ["media_item", "local_data"]
use Pinchflat.Media.MediaQuery
require Logger
alias Pinchflat.Repo
alias Pinchflat.Media
@doc """
@@ -20,14 +23,53 @@ defmodule Pinchflat.Downloading.MediaRetentionWorker do
"""
@impl Oban.Worker
def perform(%Oban.Job{}) do
cullable_media = Media.list_cullable_media_items()
cull_cullable_media_items()
delete_media_items_from_before_cutoff()
:ok
end
defp cull_cullable_media_items do
cullable_media =
MediaQuery.new()
|> MediaQuery.require_assoc(:source)
|> where(^MediaQuery.cullable())
|> Repo.all()
Logger.info("Culling #{length(cullable_media)} media items past their retention date")
Enum.each(cullable_media, fn media_item ->
# Setting `prevent_download` does what it says on the tin, but `culled_at` is purely informational.
# We don't actually do anything with that in terms of queries and it gets set to nil if the media item
# gets re-downloaded.
Media.delete_media_files(media_item, %{
prevent_download: true,
culled_at: DateTime.utc_now()
})
end)
end
# NOTE: Since this is a date and not a datetime, we can't add logic to have to-the-minute
# comparison like we can with retention periods. We can only compare to the day.
defp delete_media_items_from_before_cutoff do
deletable_media =
MediaQuery.new()
|> MediaQuery.require_assoc(:source)
|> where(^MediaQuery.deletable_based_on_source_cutoff())
|> Repo.all()
Logger.info("Deleting #{length(deletable_media)} media items that are from before the source cutoff")
Enum.each(deletable_media, fn media_item ->
# Note that I'm not setting `prevent_download` on the media_item here.
# That's because cutoff_date can easily change and it's a valid behavior to re-download older
# media items if the cutoff_date changes.
# Download is ultimately prevented because `MediaQuery.pending()` only returns media items
# from after the cutoff date (among other things), so it's not like the media will just immediately
# be re-downloaded.
Media.delete_media_files(media_item, %{
culled_at: DateTime.utc_now()
})
end)
end
end
@@ -0,0 +1,66 @@
defmodule Pinchflat.Downloading.QualityOptionBuilder do
@moduledoc """
A standalone builder module for building quality-related options for yt-dlp to download media.
Currently exclusively used in DownloadOptionBuilder since this logic is too complex to just
place in the main module.
"""
alias Pinchflat.Settings
alias Pinchflat.Profiles.MediaProfile
@doc """
Builds the quality-related options for yt-dlp to download media based on the given media profile
Includes things like container, preferred format/codec, and audio track options.
"""
def build(%MediaProfile{preferred_resolution: :audio, media_container: container} = media_profile) do
acodec = Settings.get!(:audio_codec_preference)
[
:extract_audio,
format_sort: "+acodec:#{acodec}",
audio_format: container || "best",
format: build_format_string(media_profile)
]
end
def build(%MediaProfile{preferred_resolution: resolution_atom, media_container: container} = media_profile) do
vcodec = Settings.get!(:video_codec_preference)
acodec = Settings.get!(:audio_codec_preference)
{resolution_string, _} = resolution_atom |> Atom.to_string() |> Integer.parse()
[
# Since Plex doesn't support reading metadata from MKV
remux_video: container || "mp4",
format_sort: "res:#{resolution_string},+codec:#{vcodec}:#{acodec}",
format: build_format_string(media_profile)
]
end
defp build_format_string(%MediaProfile{preferred_resolution: :audio, audio_track: audio_track}) do
if audio_track do
"bestaudio[#{build_format_modifier(audio_track)}]/bestaudio/best"
else
"bestaudio/best"
end
end
defp build_format_string(%MediaProfile{audio_track: audio_track}) do
if audio_track do
"bestvideo+bestaudio[#{build_format_modifier(audio_track)}]/bestvideo*+bestaudio/best"
else
"bestvideo*+bestaudio/best"
end
end
# Reminder to self: this conflicts with `--extractor-args "youtube:lang=<LANG>"`
# since that will translate the format_notes as well, which means they may not match.
# At least that's what happens now - worth a re-check if I have to come back to this
defp build_format_modifier("original"), do: "format_note*=original"
defp build_format_modifier("default"), do: "format_note*='(default)'"
# This uses the carat to anchor the language to the beginning of the string
# since that's what's needed to match `en` to `en-US` and `en-GB`, etc. The user
# can always specify the full language code if they want.
defp build_format_modifier(language_code), do: "language^=#{language_code}"
end
@@ -11,13 +11,27 @@ defmodule Pinchflat.FastIndexing.FastIndexingHelpers do
alias Pinchflat.Repo
alias Pinchflat.Media
alias Pinchflat.Tasks
alias Pinchflat.Sources.Source
alias Pinchflat.FastIndexing.YoutubeRss
alias Pinchflat.FastIndexing.YoutubeApi
alias Pinchflat.Downloading.DownloadingHelpers
alias Pinchflat.FastIndexing.FastIndexingWorker
alias Pinchflat.Downloading.DownloadOptionBuilder
alias Pinchflat.YtDlp.Media, as: YtDlpMedia
@doc """
Kicks off a new fast indexing task for a source. This will delete any existing fast indexing
tasks for the source before starting a new one.
Returns {:ok, %Task{}}
"""
def kickoff_indexing_task(%Source{} = source) do
Tasks.delete_pending_tasks_for(source, "FastIndexingWorker", include_executing: true)
FastIndexingWorker.kickoff_with_task(source)
end
@doc """
Fetches new media IDs for a source from YT's API or RSS, indexes them, and kicks off downloading
tasks for any pending media items. See comments in `FastIndexingWorker` for more info on the
@@ -27,6 +41,10 @@ defmodule Pinchflat.FastIndexing.FastIndexingHelpers do
downloaded_.
"""
def kickoff_download_tasks_from_youtube_rss_feed(%Source{} = source) do
# The media_profile is needed to determine the quality options to _then_ determine a more
# accurate predicted filepath
source = Repo.preload(source, [:media_profile])
{:ok, media_ids} = get_recent_media_ids(source)
existing_media_items = list_media_items_by_media_id_for(source, media_ids)
new_media_ids = media_ids -- Enum.map(existing_media_items, & &1.media_id)
@@ -68,7 +86,11 @@ defmodule Pinchflat.FastIndexing.FastIndexingHelpers do
defp create_media_item_from_media_id(source, media_id) do
url = "https://www.youtube.com/watch?v=#{media_id}"
case YtDlpMedia.get_media_attributes(url) do
command_opts =
[output: DownloadOptionBuilder.build_output_path_for(source)] ++
DownloadOptionBuilder.build_quality_options_for(source)
case YtDlpMedia.get_media_attributes(url, command_opts, use_cookies: source.use_cookies) do
{:ok, media_attrs} ->
Media.create_media_item_from_backend_attrs(source, media_attrs)
@@ -12,6 +12,8 @@ defmodule Pinchflat.Lifecycle.UserScripts.CommandRunner do
@behaviour UserScriptCommandRunner
@event_types [
:app_init,
:media_pre_download,
:media_downloaded,
:media_deleted
]
@@ -22,24 +24,25 @@ defmodule Pinchflat.Lifecycle.UserScripts.CommandRunner do
This function will succeed in almost all cases, even if the user script command
failed - this is because I don't want bad scripts to stop the whole process.
If something fails, it'll be logged.
If something fails, it'll be logged and returned BUT the tuple will always
start with {:ok, ...}.
The only things that can cause a true failure are passing in an invalid event
type or if the passed data cannot be encoded into JSON - both indicative of
failures in the development process.
Returns :ok
Returns {:ok, :no_executable} | {:ok, output, exit_code}
"""
@impl UserScriptCommandRunner
def run(event_type, encodable_data) when event_type in @event_types do
case backend_executable() do
{:ok, :no_executable} ->
:ok
{:ok, :no_executable}
{:ok, executable_path} ->
{:ok, encoded_data} = Phoenix.json_library().encode(encodable_data)
{_output, _exit_code} =
{output, exit_code} =
CliUtils.wrap_cmd(
executable_path,
[to_string(event_type), encoded_data],
@@ -47,7 +50,7 @@ defmodule Pinchflat.Lifecycle.UserScripts.CommandRunner do
logging_arg_override: "[suppressed]"
)
:ok
{:ok, output, exit_code}
end
end
@@ -62,7 +65,7 @@ defmodule Pinchflat.Lifecycle.UserScripts.CommandRunner do
if FilesystemUtils.exists_and_nonempty?(filepath) do
{:ok, filepath}
else
Logger.warning("User scripts lifecyle file either not present or is empty. Skipping.")
Logger.info("User scripts lifecyle file either not present or is empty. Skipping.")
{:ok, :no_executable}
end
+93
View File
@@ -0,0 +1,93 @@
defmodule Pinchflat.Media.FileSyncing do
@moduledoc """
Functions for ensuring file state is accurately reflected in the database.
"""
alias Pinchflat.Media
alias Pinchflat.Utils.MapUtils
alias Pinchflat.Media.MediaItem
alias Pinchflat.Utils.FilesystemUtils, as: FSUtils
@doc """
Deletes files that are no longer needed by a media item.
This means that if a media item has been updated, the old and new versions
can be passed and any files that are no longer needed will be deleted.
An example is a video that gets its quality upgraded and its name changes
between original download and re-download. The old file will exist on-disk
with the old name but the database entry will point to the new file. This
function can be used to delete the old file in this case.
Returns :ok
"""
def delete_outdated_files(old_media_item, new_media_item) do
non_subtitle_keys = MediaItem.filepath_attributes() -- [:subtitle_filepaths]
old_non_subtitles = Map.take(old_media_item, non_subtitle_keys)
old_subtitles = MapUtils.from_nested_list(old_media_item.subtitle_filepaths)
new_non_subtitles = Map.take(new_media_item, non_subtitle_keys)
new_subtitles = MapUtils.from_nested_list(new_media_item.subtitle_filepaths)
handle_file_deletion(old_non_subtitles, new_non_subtitles)
handle_file_deletion(old_subtitles, new_subtitles)
:ok
end
@doc """
Nillifies any media item filepaths that don't exist on disk for a list of media items
returns [%MediaItem{}]
"""
def sync_file_presence_on_disk(media_items) do
Enum.map(media_items, fn media_item ->
new_attributes = sync_media_item_files(media_item)
# Doing this one-by-one instead of batching since this process
# can take time and a batch could let MediaItem state get out of sync
{:ok, updated_media_item} = Media.update_media_item(media_item, new_attributes)
updated_media_item
end)
end
defp handle_file_deletion(old_attributes, new_attributes) do
# The logic:
# - A file should only be deleted if it exists and the new file is different
# - The new attributes are the ones we're interested in keeping
# - If the old attributes have a key that doesn't exist in the new attributes, don't touch it.
# This is good for archiving but may be unpopular for other users so this may change.
Enum.each(new_attributes, fn {key, new_filepath} ->
old_filepath = Map.get(old_attributes, key)
files_have_changed = old_filepath && new_filepath && old_filepath != new_filepath
files_exist_on_disk = files_have_changed && File.exists?(old_filepath) && File.exists?(new_filepath)
if files_exist_on_disk && !FSUtils.filepaths_reference_same_file?(old_filepath, new_filepath) do
FSUtils.delete_file_and_remove_empty_directories(old_filepath)
end
end)
end
defp sync_media_item_files(media_item) do
non_subtitle_keys = MediaItem.filepath_attributes() -- [:subtitle_filepaths]
subtitle_keys = MapUtils.from_nested_list(media_item.subtitle_filepaths)
non_subtitles = Map.take(media_item, non_subtitle_keys)
# This one is checking for the negative (ie: only update if the file doesn't exist)
new_non_subtitle_attrs =
Enum.reduce(non_subtitles, %{}, fn {key, filepath}, acc ->
if filepath && File.exists?(filepath), do: acc, else: Map.put(acc, key, nil)
end)
# This one is checking for the positive (ie: only update if the file exists)
# This is because subtitles, being an array type in the DB, are most easily updated
# by a full replacement rather than finding the actual diff
new_subtitle_attrs =
Enum.reduce(subtitle_keys, [], fn {key, filepath}, acc ->
if filepath && File.exists?(filepath), do: acc ++ [[key, filepath]], else: acc
end)
Map.put(new_non_subtitle_attrs, :subtitle_filepaths, new_subtitle_attrs)
end
end
@@ -0,0 +1,38 @@
defmodule Pinchflat.Media.FileSyncingWorker do
@moduledoc false
use Oban.Worker,
queue: :local_data,
tags: ["sources", "local_data"]
alias __MODULE__
alias Pinchflat.Repo
alias Pinchflat.Tasks
alias Pinchflat.Sources
alias Pinchflat.Media.FileSyncing
@doc """
Starts the source file syncing worker.
Returns {:ok, %Task{}} | {:error, %Ecto.Changeset{}}
"""
def kickoff_with_task(source, opts \\ []) do
%{id: source.id}
|> FileSyncingWorker.new(opts)
|> Tasks.create_job_with_task(source)
end
@doc """
Deletes a profile and optionally deletes its files
Returns :ok
"""
@impl Oban.Worker
def perform(%Oban.Job{args: %{"id" => source_id}}) do
source = Repo.preload(Sources.get_source!(source_id), :media_items)
FileSyncing.sync_file_presence_on_disk(source.media_items)
:ok
end
end
+15 -23
View File
@@ -15,6 +15,9 @@ defmodule Pinchflat.Media do
alias Pinchflat.Lifecycle.UserScripts.CommandRunner, as: UserScriptRunner
# Some fields should only be set on insert and not on update.
@fields_to_drop_on_update [:playlist_index]
@doc """
Returns the list of media_items.
@@ -25,21 +28,10 @@ defmodule Pinchflat.Media do
end
@doc """
Returns a list of media_items that are cullable based on the retention period
of the source they belong to.
Returns [%MediaItem{}, ...]
"""
def list_cullable_media_items do
MediaQuery.new()
|> MediaQuery.require_assoc(:source)
|> where(^MediaQuery.cullable())
|> Repo.all()
end
@doc """
Returns a list of media_items that are redownloadable based on the redownload delay
of the media_profile their source belongs to.
Returns a list of media_items that are upgradeable based on the redownload delay
of the media_profile their source belongs to. In this context, upgradeable means
that it's been long enough since upload that the video may be in a higher quality
or have better sponsorblock segments (or similar).
The logic is that a media_item is past_redownload_delay if the media_item's uploaded_at is
at least redownload_delay_days ago AND `media_downloaded_at` - `redownload_delay_days`
@@ -52,10 +44,10 @@ defmodule Pinchflat.Media do
Returns [%MediaItem{}, ...]
"""
def list_redownloadable_media_items do
def list_upgradeable_media_items do
MediaQuery.new()
|> MediaQuery.require_assoc(:media_profile)
|> where(^MediaQuery.redownloadable())
|> where(^MediaQuery.upgradeable())
|> Repo.all()
end
@@ -142,8 +134,6 @@ defmodule Pinchflat.Media do
"""
def create_media_item_from_backend_attrs(source, media_attrs_struct) do
attrs = Map.merge(%{source_id: source.id}, Map.from_struct(media_attrs_struct))
# Some fields should only be set on insert and not on update.
fields_to_drop_on_update = [:playlist_index]
%MediaItem{}
|> MediaItem.changeset(attrs)
@@ -151,7 +141,7 @@ defmodule Pinchflat.Media do
on_conflict: [
set:
attrs
|> Map.drop(fields_to_drop_on_update)
|> Map.drop(@fields_to_drop_on_update)
|> Map.to_list()
],
conflict_target: [:source_id, :media_id]
@@ -164,8 +154,10 @@ defmodule Pinchflat.Media do
Returns {:ok, %MediaItem{}} | {:error, %Ecto.Changeset{}}
"""
def update_media_item(%MediaItem{} = media_item, attrs) do
update_attrs = Map.drop(attrs, @fields_to_drop_on_update)
media_item
|> MediaItem.changeset(attrs)
|> MediaItem.changeset(update_attrs)
|> Repo.update()
end
@@ -182,7 +174,7 @@ defmodule Pinchflat.Media do
if delete_files do
{:ok, _} = do_delete_media_files(media_item)
:ok = run_user_script(:media_deleted, media_item)
run_user_script(:media_deleted, media_item)
end
# Should delete these no matter what
@@ -205,7 +197,7 @@ defmodule Pinchflat.Media do
Tasks.delete_tasks_for(media_item)
{:ok, _} = do_delete_media_files(media_item)
:ok = run_user_script(:media_deleted, media_item)
run_user_script(:media_deleted, media_item)
update_media_item(media_item, Map.merge(filepath_attrs, addl_attrs))
end
+2
View File
@@ -31,6 +31,7 @@ defmodule Pinchflat.Media.MediaItem do
:uploaded_at,
:upload_date_index,
:duration_seconds,
:predicted_media_filepath,
# these fields are captured only on download
:media_downloaded_at,
:media_filepath,
@@ -76,6 +77,7 @@ defmodule Pinchflat.Media.MediaItem do
field :duration_seconds, :integer
field :playlist_index, :integer, default: 0
field :predicted_media_filepath, :string
field :media_filepath, :string
field :media_size_bytes, :integer
field :thumbnail_filepath, :string
+23 -7
View File
@@ -33,7 +33,6 @@ defmodule Pinchflat.Media.MediaQuery do
def downloaded, do: dynamic([mi], not is_nil(mi.media_filepath))
def download_prevented, do: dynamic([mi], mi.prevent_download == true)
def culling_prevented, do: dynamic([mi], mi.prevent_culling == true)
def culled, do: dynamic([mi], not is_nil(mi.culled_at))
def redownloaded, do: dynamic([mi], not is_nil(mi.media_redownloaded_at))
def upload_date_matches(other_date), do: dynamic([mi], fragment("date(?) = date(?)", mi.uploaded_at, ^other_date))
@@ -76,12 +75,20 @@ defmodule Pinchflat.Media.MediaQuery do
)
end
def meets_min_and_max_duration do
dynamic(
[mi, source],
(is_nil(source.min_duration_seconds) or fragment("duration_seconds >= ?", source.min_duration_seconds)) and
(is_nil(source.max_duration_seconds) or fragment("duration_seconds <= ?", source.max_duration_seconds))
)
end
def past_retention_period do
dynamic(
[mi, source],
fragment("""
IFNULL(retention_period_days, 0) > 0 AND
DATETIME('now', '-' || retention_period_days || ' day') > media_downloaded_at
DATETIME(media_downloaded_at, '+' || retention_period_days || ' day') < DATETIME('now')
""")
)
end
@@ -93,8 +100,8 @@ defmodule Pinchflat.Media.MediaQuery do
# downloaded_at minus the redownload_delay_days is before the upload date
fragment("""
IFNULL(redownload_delay_days, 0) > 0 AND
DATETIME('now', '-' || redownload_delay_days || ' day') > uploaded_at AND
DATETIME(media_downloaded_at, '-' || redownload_delay_days || ' day') < uploaded_at
DATE('now', '-' || redownload_delay_days || ' day') > DATE(uploaded_at) AND
DATE(media_downloaded_at, '-' || redownload_delay_days || ' day') < DATE(uploaded_at)
""")
)
end
@@ -108,6 +115,15 @@ defmodule Pinchflat.Media.MediaQuery do
)
end
def deletable_based_on_source_cutoff do
dynamic(
[mi, source],
^downloaded() and
not (^upload_date_after_source_cutoff()) and
not (^culling_prevented())
)
end
def pending do
dynamic(
[mi],
@@ -115,16 +131,16 @@ defmodule Pinchflat.Media.MediaQuery do
not (^download_prevented()) and
^upload_date_after_source_cutoff() and
^format_matching_profile_preference() and
^matches_source_title_regex()
^matches_source_title_regex() and
^meets_min_and_max_duration()
)
end
def redownloadable do
def upgradeable do
dynamic(
[mi, source],
^downloaded() and
not (^download_prevented()) and
not (^culled()) and
not (^redownloaded()) and
^past_redownload_delay()
)
@@ -62,11 +62,13 @@ defmodule Pinchflat.Metadata.MetadataFileHelpers do
Returns binary() | nil
"""
def download_and_store_thumbnail_for(database_record) do
yt_dlp_filepath = generate_filepath_for(database_record, "thumbnail.%(ext)s")
real_filepath = generate_filepath_for(database_record, "thumbnail.jpg")
def download_and_store_thumbnail_for(media_item_with_preloads) do
yt_dlp_filepath = generate_filepath_for(media_item_with_preloads, "thumbnail.%(ext)s")
real_filepath = generate_filepath_for(media_item_with_preloads, "thumbnail.jpg")
command_opts = [output: yt_dlp_filepath]
addl_opts = [use_cookies: media_item_with_preloads.source.use_cookies]
case YtDlpMedia.download_thumbnail(database_record.original_url, output: yt_dlp_filepath) do
case YtDlpMedia.download_thumbnail(media_item_with_preloads.original_url, command_opts, addl_opts) do
{:ok, _} -> real_filepath
_ -> nil
end
@@ -92,7 +92,9 @@ defmodule Pinchflat.Metadata.SourceMetadataStorageWorker do
defp determine_series_directory(source) do
output_path = DownloadOptionBuilder.build_output_path_for(source)
{:ok, %{filepath: filepath}} = MediaCollection.get_source_details(source.original_url, output: output_path)
runner_opts = [output: output_path]
addl_opts = [use_cookies: source.use_cookies]
{:ok, %{filepath: filepath}} = MediaCollection.get_source_details(source.original_url, runner_opts, addl_opts)
case MetadataFileHelpers.series_directory_from_media_filepath(filepath) do
{:ok, series_directory} -> series_directory
@@ -119,7 +121,7 @@ defmodule Pinchflat.Metadata.SourceMetadataStorageWorker do
base_opts ++ [:write_thumbnail, playlist_items: 1]
end
MediaCollection.get_source_metadata(source.original_url, opts)
MediaCollection.get_source_metadata(source.original_url, opts, use_cookies: source.use_cookies)
end
defp tmp_directory do
+7
View File
@@ -26,8 +26,11 @@ defmodule Pinchflat.Profiles.MediaProfile do
sponsorblock_categories
shorts_behaviour
livestream_behaviour
audio_track
preferred_resolution
media_container
redownload_delay_days
marked_for_deletion_at
)a
@required_fields ~w(name output_path_template)a
@@ -63,7 +66,11 @@ defmodule Pinchflat.Profiles.MediaProfile do
# See `build_format_clauses` in the Media context for more.
field :shorts_behaviour, Ecto.Enum, values: ~w(include exclude only)a, default: :include
field :livestream_behaviour, Ecto.Enum, values: ~w(include exclude only)a, default: :include
field :audio_track, :string
field :preferred_resolution, Ecto.Enum, values: ~w(4320p 2160p 1080p 720p 480p 360p audio)a, default: :"1080p"
field :media_container, :string, default: nil
field :marked_for_deletion_at, :utc_datetime
has_many :sources, Source
@@ -0,0 +1,38 @@
defmodule Pinchflat.Profiles.MediaProfileDeletionWorker do
@moduledoc false
use Oban.Worker,
queue: :local_data,
tags: ["media_profiles", "local_data"]
require Logger
alias __MODULE__
alias Pinchflat.Profiles
@doc """
Starts the profile deletion worker. Does not attach it to a task like `kickoff_with_task/2`
since deletion also cancels all tasks for the profile
Returns {:ok, %Task{}} | {:error, %Ecto.Changeset{}}
"""
def kickoff(profile, job_args \\ %{}, job_opts \\ []) do
%{id: profile.id}
|> Map.merge(job_args)
|> MediaProfileDeletionWorker.new(job_opts)
|> Oban.insert()
end
@doc """
Deletes a profile and optionally deletes its files
Returns :ok
"""
@impl Oban.Worker
def perform(%Oban.Job{args: %{"id" => profile_id} = args}) do
delete_files = Map.get(args, "delete_files", false)
profile = Profiles.get_media_profile!(profile_id)
Profiles.delete_media_profile(profile, delete_files: delete_files)
end
end
+29
View File
@@ -0,0 +1,29 @@
defmodule Pinchflat.Profiles.ProfilesQuery do
@moduledoc """
Query helpers for the Profiles context.
These methods are made to be one-ish liners used
to compose queries. Each method should strive to do
_one_ thing. These don't need to be tested as
they are just building blocks for other functionality
which, itself, will be tested.
"""
import Ecto.Query, warn: false
alias Pinchflat.Profiles.MediaProfile
# This allows the module to be aliased and query methods to be used
# all in one go
# usage: use Pinchflat.Profiles.ProfilesQuery
defmacro __using__(_opts) do
quote do
import Ecto.Query, warn: false
alias unquote(__MODULE__)
end
end
def new do
MediaProfile
end
end
+2
View File
@@ -29,6 +29,8 @@ defmodule Pinchflat.Release do
[
"/config",
"/downloads",
"/etc/yt-dlp",
"/etc/yt-dlp/plugins",
Application.get_env(:pinchflat, :media_directory),
Application.get_env(:pinchflat, :tmpfile_directory),
Application.get_env(:pinchflat, :extras_directory),
@@ -16,6 +16,7 @@ defmodule Pinchflat.SlowIndexing.SlowIndexingHelpers do
alias Pinchflat.YtDlp.MediaCollection
alias Pinchflat.Downloading.DownloadingHelpers
alias Pinchflat.SlowIndexing.FileFollowerServer
alias Pinchflat.Downloading.DownloadOptionBuilder
alias Pinchflat.SlowIndexing.MediaCollectionIndexingWorker
alias Pinchflat.YtDlp.Media, as: YtDlpMedia
@@ -24,13 +25,28 @@ defmodule Pinchflat.SlowIndexing.SlowIndexingHelpers do
Starts tasks for indexing a source's media regardless of the source's indexing
frequency. It's assumed the caller will check for indexing frequency.
Returns {:ok, %Task{}}.
Returns {:ok, %Task{}}
"""
def kickoff_indexing_task(%Source{} = source, job_args \\ %{}, job_opts \\ []) do
Tasks.delete_pending_tasks_for(source, "FastIndexingWorker")
Tasks.delete_pending_tasks_for(source, "MediaCollectionIndexingWorker")
job_offset_seconds = calculate_job_offset_seconds(source)
MediaCollectionIndexingWorker.kickoff_with_task(source, job_args, job_opts)
Tasks.delete_pending_tasks_for(source, "FastIndexingWorker")
Tasks.delete_pending_tasks_for(source, "MediaCollectionIndexingWorker", include_executing: true)
MediaCollectionIndexingWorker.kickoff_with_task(source, job_args, job_opts ++ [schedule_in: job_offset_seconds])
end
@doc """
A helper method to delete all indexing-related tasks for a source.
Optionally, you can include executing tasks in the deletion process.
Returns :ok
"""
def delete_indexing_tasks(%Source{} = source, opts \\ []) do
include_executing = Keyword.get(opts, :include_executing, false)
Tasks.delete_pending_tasks_for(source, "FastIndexingWorker", include_executing: include_executing)
Tasks.delete_pending_tasks_for(source, "MediaCollectionIndexingWorker", include_executing: include_executing)
end
@doc """
@@ -56,6 +72,9 @@ defmodule Pinchflat.SlowIndexing.SlowIndexingHelpers do
Returns [%MediaItem{} | %Ecto.Changeset{}]
"""
def index_and_enqueue_download_for_media_items(%Source{} = source) do
# The media_profile is needed to determine the quality options to _then_ determine a more
# accurate predicted filepath
source = Repo.preload(source, [:media_profile])
# See the method definition below for more info on how file watchers work
# (important reading if you're not familiar with it)
{:ok, media_attributes} = setup_file_watcher_and_kickoff_indexing(source)
@@ -94,7 +113,13 @@ defmodule Pinchflat.SlowIndexing.SlowIndexingHelpers do
{:ok, pid} = FileFollowerServer.start_link()
handler = fn filepath -> setup_file_follower_watcher(pid, filepath, source) end
result = MediaCollection.get_media_attributes_for_collection(source.original_url, file_listener_handler: handler)
command_opts =
[output: DownloadOptionBuilder.build_output_path_for(source)] ++
DownloadOptionBuilder.build_quality_options_for(source)
runner_opts = [file_listener_handler: handler, use_cookies: source.use_cookies]
result = MediaCollection.get_media_attributes_for_collection(source.original_url, command_opts, runner_opts)
FileFollowerServer.stop(pid)
@@ -131,4 +156,14 @@ defmodule Pinchflat.SlowIndexing.SlowIndexingHelpers do
changeset
end
end
# Find the difference between the current time and the last time the source was indexed
defp calculate_job_offset_seconds(%Source{last_indexed_at: nil}), do: 0
defp calculate_job_offset_seconds(source) do
offset_seconds = DateTime.diff(DateTime.utc_now(), source.last_indexed_at, :second)
index_frequency_seconds = source.index_frequency_minutes * 60
max(0, index_frequency_seconds - offset_seconds)
end
end
+24
View File
@@ -15,6 +15,7 @@ defmodule Pinchflat.Sources.Source do
alias Pinchflat.Metadata.SourceMetadata
@allowed_fields ~w(
enabled
collection_name
collection_id
collection_type
@@ -27,6 +28,7 @@ defmodule Pinchflat.Sources.Source do
series_directory
index_frequency_minutes
fast_index
use_cookies
download_media
last_indexed_at
original_url
@@ -35,6 +37,9 @@ defmodule Pinchflat.Sources.Source do
title_filter_regex
media_profile_id
output_path_template_override
marked_for_deletion_at
min_duration_seconds
max_duration_seconds
)a
# Expensive API calls are made when a source is inserted/updated so
@@ -60,6 +65,7 @@ defmodule Pinchflat.Sources.Source do
)a
schema "sources" do
field :enabled, :boolean, default: true
# This is _not_ used as the primary key or internally in the database
# relations. This is only used to prevent an enumeration attack on the streaming
# and RSS feed endpoints since those _must_ be public (ie: no basic auth)
@@ -72,6 +78,7 @@ defmodule Pinchflat.Sources.Source do
field :collection_type, Ecto.Enum, values: [:channel, :playlist]
field :index_frequency_minutes, :integer, default: 60 * 24
field :fast_index, :boolean, default: false
field :use_cookies, :boolean, default: false
field :download_media, :boolean, default: true
field :last_indexed_at, :utc_datetime
# Only download media items that were published after this date
@@ -81,12 +88,17 @@ defmodule Pinchflat.Sources.Source do
field :title_filter_regex, :string
field :output_path_template_override, :string
field :min_duration_seconds, :integer
field :max_duration_seconds, :integer
field :series_directory, :string
field :nfo_filepath, :string
field :poster_filepath, :string
field :fanart_filepath, :string
field :banner_filepath, :string
field :marked_for_deletion_at, :utc_datetime
belongs_to :media_profile, MediaProfile
has_one :metadata, SourceMetadata, on_replace: :update
@@ -113,6 +125,7 @@ defmodule Pinchflat.Sources.Source do
|> dynamic_default(:uuid, fn _ -> Ecto.UUID.generate() end)
|> validate_required(required_fields)
|> validate_title_regex()
|> validate_min_and_max_durations()
|> validate_number(:retention_period_days, greater_than_or_equal_to: 0)
# Ensures it ends with `.{{ ext }}` or `.%(ext)s` or similar (with a little wiggle room)
|> validate_format(:output_path_template_override, MediaProfile.ext_regex(), message: "must end with .{{ ext }}")
@@ -159,6 +172,17 @@ defmodule Pinchflat.Sources.Source do
defp validate_title_regex(changeset), do: changeset
defp validate_min_and_max_durations(changeset) do
min_duration = get_change(changeset, :min_duration_seconds)
max_duration = get_change(changeset, :max_duration_seconds)
case {min_duration, max_duration} do
{min, max} when is_nil(min) or is_nil(max) -> changeset
{min, max} when min >= max -> add_error(changeset, :max_duration_seconds, "must be greater than minumum duration")
_ -> changeset
end
end
defimpl Jason.Encoder, for: Source do
def encode(value, opts) do
value
@@ -0,0 +1,38 @@
defmodule Pinchflat.Sources.SourceDeletionWorker do
@moduledoc false
use Oban.Worker,
queue: :local_data,
tags: ["sources", "local_data"]
require Logger
alias __MODULE__
alias Pinchflat.Sources
@doc """
Starts the source deletion worker. Does not attach it to a task like `kickoff_with_task/2`
since deletion also cancels all tasks for the source
Returns {:ok, %Task{}} | {:error, %Ecto.Changeset{}}
"""
def kickoff(source, job_args \\ %{}, job_opts \\ []) do
%{id: source.id}
|> Map.merge(job_args)
|> SourceDeletionWorker.new(job_opts)
|> Oban.insert()
end
@doc """
Deletes a source and optionally deletes its files
Returns :ok
"""
@impl Oban.Worker
def perform(%Oban.Job{args: %{"id" => source_id} = args}) do
delete_files = Map.get(args, "delete_files", false)
source = Sources.get_source!(source_id)
Sources.delete_source(source, delete_files: delete_files)
end
end
+69 -19
View File
@@ -15,8 +15,8 @@ defmodule Pinchflat.Sources do
alias Pinchflat.Metadata.SourceMetadata
alias Pinchflat.Utils.FilesystemUtils
alias Pinchflat.Downloading.DownloadingHelpers
alias Pinchflat.FastIndexing.FastIndexingWorker
alias Pinchflat.SlowIndexing.SlowIndexingHelpers
alias Pinchflat.FastIndexing.FastIndexingHelpers
alias Pinchflat.Metadata.SourceMetadataStorageWorker
@doc """
@@ -180,11 +180,19 @@ defmodule Pinchflat.Sources do
end
defp add_source_details_to_changeset(source, changeset) do
case MediaCollection.get_source_details(changeset.changes.original_url) do
use_cookies = Ecto.Changeset.get_field(changeset, :use_cookies)
case MediaCollection.get_source_details(changeset.changes.original_url, [], use_cookies: use_cookies) do
{:ok, source_details} ->
add_source_details_by_collection_type(source, changeset, source_details)
{:error, runner_error, _status_code} ->
err ->
runner_error =
case err do
{:error, error_msg, _status_code} -> error_msg
{:error, error_msg} -> error_msg
end
Ecto.Changeset.add_error(
changeset,
:original_url,
@@ -247,19 +255,40 @@ defmodule Pinchflat.Sources do
end
end
# If the source is NOT new (ie: updated) and the download_media flag has changed,
# If the source is new (ie: not persisted), do nothing
defp maybe_handle_media_tasks(%{data: %{__meta__: %{state: state}}}, _source) when state != :loaded do
:ok
end
# If the source is NOT new (ie: updated),
# enqueue or dequeue media download tasks as necessary.
defp maybe_handle_media_tasks(changeset, source) do
case {changeset.data, changeset.changes} do
{%{__meta__: %{state: :loaded}}, %{download_media: true}} ->
current_changes = changeset.changes
applied_changes = Ecto.Changeset.apply_changes(changeset)
# We need both current_changes and applied_changes to determine
# the course of action to take. For example, we only care if a source is supposed
# to be `enabled` or not - we don't care if that information comes from the
# current changes or if that's how it already was in the database.
# Rephrased, we're essentially using it in place of `get_field/2`
case {current_changes, applied_changes} do
{%{download_media: true}, %{enabled: true}} ->
DownloadingHelpers.enqueue_pending_download_tasks(source)
{%{__meta__: %{state: :loaded}}, %{download_media: false}} ->
{%{enabled: true}, %{download_media: true}} ->
DownloadingHelpers.enqueue_pending_download_tasks(source)
{%{download_media: false}, _} ->
DownloadingHelpers.dequeue_pending_download_tasks(source)
{%{enabled: false}, _} ->
DownloadingHelpers.dequeue_pending_download_tasks(source)
_ ->
:ok
nil
end
:ok
end
defp maybe_run_indexing_task(changeset, source) do
@@ -293,13 +322,22 @@ defmodule Pinchflat.Sources do
end
defp maybe_update_slow_indexing_task(changeset, source) do
case changeset.changes do
%{index_frequency_minutes: mins} when mins > 0 ->
# See comment in `maybe_handle_media_tasks` as to why we need these
current_changes = changeset.changes
applied_changes = Ecto.Changeset.apply_changes(changeset)
case {current_changes, applied_changes} do
{%{index_frequency_minutes: mins}, %{enabled: true}} when mins > 0 ->
SlowIndexingHelpers.kickoff_indexing_task(source)
%{index_frequency_minutes: _} ->
Tasks.delete_pending_tasks_for(source, "FastIndexingWorker")
Tasks.delete_pending_tasks_for(source, "MediaCollectionIndexingWorker")
{%{enabled: true}, %{index_frequency_minutes: mins}} when mins > 0 ->
SlowIndexingHelpers.kickoff_indexing_task(source)
{%{index_frequency_minutes: _}, _} ->
SlowIndexingHelpers.delete_indexing_tasks(source, include_executing: true)
{%{enabled: false}, _} ->
SlowIndexingHelpers.delete_indexing_tasks(source, include_executing: true)
_ ->
:ok
@@ -307,13 +345,25 @@ defmodule Pinchflat.Sources do
end
defp maybe_update_fast_indexing_task(changeset, source) do
case changeset.changes do
%{fast_index: true} ->
Tasks.delete_pending_tasks_for(source, "FastIndexingWorker")
FastIndexingWorker.kickoff_with_task(source)
# See comment in `maybe_handle_media_tasks` as to why we need these
current_changes = changeset.changes
applied_changes = Ecto.Changeset.apply_changes(changeset)
%{fast_index: false} ->
Tasks.delete_pending_tasks_for(source, "FastIndexingWorker")
# This technically could be simplified since `maybe_update_slow_indexing_task`
# has some overlap re: deleting pending tasks, but I'm keeping it separate
# for clarity and explicitness.
case {current_changes, applied_changes} do
{%{fast_index: true}, %{enabled: true}} ->
FastIndexingHelpers.kickoff_indexing_task(source)
{%{enabled: true}, %{fast_index: true}} ->
FastIndexingHelpers.kickoff_indexing_task(source)
{%{fast_index: false}, _} ->
Tasks.delete_pending_tasks_for(source, "FastIndexingWorker", include_executing: true)
{%{enabled: false}, _} ->
Tasks.delete_pending_tasks_for(source, "FastIndexingWorker", include_executing: true)
_ ->
:ok
+9 -21
View File
@@ -53,20 +53,6 @@ defmodule Pinchflat.Tasks do
)
end
@doc """
Returns the list of pending tasks for a given record type and ID. Optionally allows you to specify
which worker to include.
Returns [%Task{}, ...]
"""
def list_pending_tasks_for(record, worker_name \\ nil) do
list_tasks_for(
record,
worker_name,
[:available, :scheduled, :retryable]
)
end
@doc """
Gets a single task.
@@ -127,13 +113,13 @@ defmodule Pinchflat.Tasks do
@doc """
Deletes all tasks attached to a given record, cancelling any attached jobs.
Optionally allows you to specify which worker to include.
Optionally allows you to specify which worker and job states to include.
Returns :ok
"""
def delete_tasks_for(record, worker_name \\ nil) do
def delete_tasks_for(record, worker_name \\ nil, job_states \\ Oban.Job.states()) do
record
|> list_tasks_for(worker_name)
|> list_tasks_for(worker_name, job_states)
|> Enum.each(&delete_task/1)
end
@@ -143,10 +129,12 @@ defmodule Pinchflat.Tasks do
Returns :ok
"""
def delete_pending_tasks_for(record, worker_name \\ nil) do
record
|> list_pending_tasks_for(worker_name)
|> Enum.each(&delete_task/1)
def delete_pending_tasks_for(record, worker_name \\ nil, opts \\ []) do
include_executing = Keyword.get(opts, :include_executing, false)
base_job_states = [:available, :scheduled, :retryable]
job_states = if include_executing, do: base_job_states ++ [:executing], else: base_job_states
delete_tasks_for(record, worker_name, job_states)
end
@doc """
+18
View File
@@ -20,6 +20,24 @@ defmodule Pinchflat.Utils.FilesystemUtils do
end
end
@doc """
Checks if two filepaths reference the same file.
Useful if you have a relative and absolute filepath and want to be sure they're the same file.
Also works with symlinks.
Returns boolean()
"""
def filepaths_reference_same_file?(filepath_1, filepath_2) do
{:ok, stat_1} = File.stat(filepath_1)
{:ok, stat_2} = File.stat(filepath_2)
identifier_1 = "#{stat_1.major_device}:#{stat_1.minor_device}:#{stat_1.inode}"
identifier_2 = "#{stat_2.major_device}:#{stat_2.minor_device}:#{stat_2.inode}"
identifier_1 == identifier_2
end
@doc """
Generates a temporary file and returns its path. The file is empty and has the given type.
Generates all the directories in the path if they don't exist.
+17
View File
@@ -0,0 +1,17 @@
defmodule Pinchflat.Utils.MapUtils do
@moduledoc """
Utility methods for working with maps
"""
@doc """
Converts a nested list of 2-element tuples or lists into a map.
Returns map()
"""
def from_nested_list(list) do
Enum.reduce(list, %{}, fn
[key, value], acc -> Map.put(acc, key, value)
{key, value}, acc -> Map.put(acc, key, value)
end)
end
end
+11 -2
View File
@@ -18,6 +18,8 @@ defmodule Pinchflat.YtDlp.CommandRunner do
- :output_filepath - the path to save the output to. If not provided, a temporary
file will be created and used. Useful for if you need a reference to the file
for a file watcher.
- :use_cookies - if true, will add a cookie file to the command options. Will not
attach a cookie file if the user hasn't set one up.
Returns {:ok, binary()} | {:error, output, status}.
"""
@@ -28,7 +30,7 @@ defmodule Pinchflat.YtDlp.CommandRunner do
output_filepath = generate_output_filepath(addl_opts)
print_to_file_opts = [{:print_to_file, output_template}, output_filepath]
user_configured_opts = cookie_file_options()
user_configured_opts = cookie_file_options(addl_opts)
# These must stay in exactly this order, hence why I'm giving it its own variable.
all_opts = command_opts ++ print_to_file_opts ++ user_configured_opts ++ global_options()
formatted_command_opts = [url] ++ CliUtils.parse_options(all_opts)
@@ -78,7 +80,14 @@ defmodule Pinchflat.YtDlp.CommandRunner do
]
end
defp cookie_file_options do
defp cookie_file_options(addl_opts) do
case Keyword.get(addl_opts, :use_cookies) do
true -> add_cookie_file()
_ -> []
end
end
defp add_cookie_file do
base_dir = Application.get_env(:pinchflat, :extras_directory)
filename_options_map = %{cookies: "cookies.txt"}
+52 -18
View File
@@ -11,7 +11,8 @@ defmodule Pinchflat.YtDlp.Media do
:livestream,
:short_form_content,
:uploaded_at,
:duration_seconds
:duration_seconds,
:predicted_media_filepath
]
defstruct [
@@ -23,7 +24,8 @@ defmodule Pinchflat.YtDlp.Media do
:short_form_content,
:uploaded_at,
:duration_seconds,
:playlist_index
:playlist_index,
:predicted_media_filepath
]
alias __MODULE__
@@ -37,9 +39,9 @@ defmodule Pinchflat.YtDlp.Media do
Returns {:ok, map()} | {:error, any, ...}.
"""
def download(url, command_opts \\ [], addl_opts \\ []) do
opts = [:no_simulate] ++ command_opts
all_command_opts = [:no_simulate] ++ command_opts
with {:ok, output} <- backend_runner().run(url, opts, "after_move:%()j", addl_opts),
with {:ok, output} <- backend_runner().run(url, all_command_opts, "after_move:%()j", addl_opts),
{:ok, parsed_json} <- Phoenix.json_library().decode(output) do
{:ok, parsed_json}
else
@@ -47,31 +49,50 @@ defmodule Pinchflat.YtDlp.Media do
end
end
@doc """
Determines if the media at the given URL is ready to be downloaded.
Common examples of non-downloadable media are upcoming or in-progress live streams.
Returns {:ok, :downloadable | :ignorable} | {:error, any}
"""
def get_downloadable_status(url) do
case backend_runner().run(url, [:simulate, :skip_download], "%(.{live_status})j") do
{:ok, output} ->
output
|> Phoenix.json_library().decode!()
|> parse_downloadable_status()
err ->
err
end
end
@doc """
Downloads a thumbnail for a single piece of media. Usually used for
downloading thumbnails for internal use
Returns {:ok, ""} | {:error, any, ...}.
"""
def download_thumbnail(url, command_opts \\ []) do
opts = [:no_simulate, :skip_download, :write_thumbnail, convert_thumbnail: "jpg"] ++ command_opts
def download_thumbnail(url, command_opts \\ [], addl_opts \\ []) do
all_command_opts = [:no_simulate, :skip_download, :write_thumbnail, convert_thumbnail: "jpg"] ++ command_opts
# NOTE: it doesn't seem like this command actually returns anything in `after_move` since
# we aren't downloading the main media file
backend_runner().run(url, opts, "after_move:%()j")
backend_runner().run(url, all_command_opts, "after_move:%()j", addl_opts)
end
@doc """
Returns a map representing the media at the given URL.
Optionally takes a list of additional command options to pass to yt-dlp
or configuration-related options to pass to the runner.
Returns {:ok, %Media{}} | {:error, any, ...}.
"""
def get_media_attributes(url) do
runner = Application.get_env(:pinchflat, :yt_dlp_runner)
command_opts = [:simulate, :skip_download]
def get_media_attributes(url, command_opts \\ [], addl_opts \\ []) do
all_command_opts = [:simulate, :skip_download] ++ command_opts
output_template = indexing_output_template()
case runner.run(url, command_opts, output_template) do
case backend_runner().run(url, all_command_opts, output_template, addl_opts) do
{:ok, output} ->
output
|> Phoenix.json_library().decode!()
@@ -87,9 +108,11 @@ defmodule Pinchflat.YtDlp.Media do
Returns the output template for yt-dlp's indexing command.
NOTE: playlist_index is really only useful for playlists that will never change their order.
NOTE: I've switched back to `original_url` (from `webpage_url`) since it's started indicating
if something is a short via the URL again
"""
def indexing_output_template do
"%(.{id,title,was_live,webpage_url,description,aspect_ratio,duration,upload_date,timestamp,playlist_index})j"
"%(.{id,title,live_status,original_url,description,aspect_ratio,duration,upload_date,timestamp,playlist_index,filename})j"
end
@doc """
@@ -103,17 +126,18 @@ defmodule Pinchflat.YtDlp.Media do
media_id: response["id"],
title: response["title"],
description: response["description"],
original_url: response["webpage_url"],
livestream: !!response["was_live"],
original_url: response["original_url"],
livestream: !!response["live_status"] && response["live_status"] != "not_live",
duration_seconds: response["duration"] && round(response["duration"]),
short_form_content: response["webpage_url"] && short_form_content?(response),
short_form_content: response["original_url"] && short_form_content?(response),
uploaded_at: response["upload_date"] && parse_uploaded_at(response),
playlist_index: response["playlist_index"] || 0
playlist_index: response["playlist_index"] || 0,
predicted_media_filepath: response["filename"]
}
end
defp short_form_content?(response) do
if String.contains?(response["webpage_url"], "/shorts/") do
if String.contains?(response["original_url"], "/shorts/") do
true
else
# Sometimes shorts are returned without /shorts/ in the URL,
@@ -124,7 +148,7 @@ defmodule Pinchflat.YtDlp.Media do
#
# These don't fail if duration or aspect_ratio are missing
# due to Elixir's comparison semantics
response["duration"] <= 60 && response["aspect_ratio"] < 0.8
response["duration"] <= 60 && response["aspect_ratio"] <= 0.85
end
end
@@ -140,6 +164,16 @@ defmodule Pinchflat.YtDlp.Media do
defp parse_uploaded_at(%{"upload_date" => nil}), do: nil
defp parse_uploaded_at(response), do: MetadataFileHelpers.parse_upload_date(response["upload_date"])
defp parse_downloadable_status(response) do
case response["live_status"] do
status when status in ["is_live", "is_upcoming", "post_live"] -> {:ok, :ignorable}
status when status in ["was_live", "not_live"] -> {:ok, :downloadable}
# This preserves my tenuous support for non-youtube sources.
nil -> {:ok, :downloadable}
_ -> {:error, "Unknown live status: #{response["live_status"]}"}
end
end
defp backend_runner do
# This approach lets us mock the command for testing
Application.get_env(:pinchflat, :yt_dlp_runner)
+17 -11
View File
@@ -11,29 +11,34 @@ defmodule Pinchflat.YtDlp.MediaCollection do
@doc """
Returns a list of maps representing the media in the collection.
Optionally takes a list of additional command options to pass to yt-dlp
or configuration-related options to pass to the runner.
Options:
Runner Options:
- :file_listener_handler - a function that will be called with the path to the
file that will be written to when yt-dlp is done. This is useful for
setting up a file watcher to know when the file is ready to be read.
- :use_cookies - whether or not to use user-provided cookies when fetching the media details
Returns {:ok, [map()]} | {:error, any, ...}.
"""
def get_media_attributes_for_collection(url, addl_opts \\ []) do
def get_media_attributes_for_collection(url, command_opts \\ [], addl_opts \\ []) do
runner = Application.get_env(:pinchflat, :yt_dlp_runner)
# `ignore_no_formats_error` is necessary because yt-dlp will error out if
# the first video has not released yet (ie: is a premier). We don't care about
# available formats since we're just getting the media details
command_opts = [:simulate, :skip_download, :ignore_no_formats_error, :no_warnings]
all_command_opts = [:simulate, :skip_download, :ignore_no_formats_error, :no_warnings] ++ command_opts
use_cookies = Keyword.get(addl_opts, :use_cookies, false)
output_template = YtDlpMedia.indexing_output_template()
output_filepath = FilesystemUtils.generate_metadata_tmpfile(:json)
file_listener_handler = Keyword.get(addl_opts, :file_listener_handler, false)
runner_opts = [output_filepath: output_filepath, use_cookies: use_cookies]
if file_listener_handler do
file_listener_handler.(output_filepath)
end
case runner.run(url, command_opts, output_template, output_filepath: output_filepath) do
case runner.run(url, all_command_opts, output_template, runner_opts) do
{:ok, output} ->
parsed_lines =
output
@@ -64,7 +69,7 @@ defmodule Pinchflat.YtDlp.MediaCollection do
Returns {:ok, map()} | {:error, any, ...}.
"""
def get_source_details(source_url, addl_opts \\ []) do
def get_source_details(source_url, command_opts \\ [], addl_opts \\ []) do
# `ignore_no_formats_error` is necessary because yt-dlp will error out if
# the first video has not released yet (ie: is a premier). We don't care about
# available formats since we're just getting the source details
@@ -75,13 +80,14 @@ defmodule Pinchflat.YtDlp.MediaCollection do
playlist_end: 1
]
command_opts = default_opts ++ addl_opts
all_command_opts = default_opts ++ command_opts
output_template = "%(.{channel,channel_id,playlist_id,playlist_title,filename})j"
with {:ok, output} <- backend_runner().run(source_url, command_opts, output_template),
with {:ok, output} <- backend_runner().run(source_url, all_command_opts, output_template, addl_opts),
{:ok, parsed_json} <- Phoenix.json_library().decode(output) do
{:ok, format_source_details(parsed_json)}
else
{:error, %Jason.DecodeError{}} -> {:error, "Error decoding JSON response"}
err -> err
end
end
@@ -109,14 +115,14 @@ defmodule Pinchflat.YtDlp.MediaCollection do
Returns {:ok, map()} | {:error, any, ...}.
"""
def get_source_metadata(source_url, addl_opts \\ [playlist_items: 0]) do
def get_source_metadata(source_url, command_opts, addl_opts \\ []) do
# This only validates that the `playlist_items` key is present. It's otherwise unused
_playlist_items = Keyword.fetch!(addl_opts, :playlist_items)
_playlist_items = Keyword.fetch!(command_opts, :playlist_items)
opts = [:skip_download] ++ addl_opts
all_command_opts = [:skip_download] ++ command_opts
output_template = "playlist:%()j"
with {:ok, output} <- backend_runner().run(source_url, opts, output_template),
with {:ok, output} <- backend_runner().run(source_url, all_command_opts, output_template, addl_opts),
{:ok, parsed_json} <- Phoenix.json_library().decode(output) do
{:ok, parsed_json}
else
@@ -340,14 +340,15 @@ defmodule PinchflatWeb.CoreComponents do
end)
~H"""
<div x-data={"{ enabled: #{@checked}}"}>
<.label for={@id}>
<div x-data={"{ enabled: #{@checked} }"} class="" phx-update="ignore" id={"#{@id}-wrapper"}>
<.label :if={@label} for={@id}>
<%= @label %>
<span :if={@label_suffix} class="text-xs text-bodydark"><%= @label_suffix %></span>
</.label>
<div class="relative">
<div class="relative flex flex-col">
<input type="hidden" id={@id} name={@name} x-bind:value="enabled" {@rest} />
<div class="inline-block cursor-pointer" @click="enabled = !enabled">
<%!-- This triggers a `change` event on the hidden input when the toggle is clicked --%>
<div class="inline-block cursor-pointer" @click={"enabled = !enabled; dispatchFor('#{@id}', 'change')"}>
<div x-bind:class="enabled && '!bg-primary'" class="block h-8 w-14 rounded-full bg-black"></div>
<div
x-bind:class="enabled && '!right-1 !translate-x-full'"
@@ -62,17 +62,24 @@ defmodule PinchflatWeb.CustomComponents.TextComponents do
end
@doc """
Renders a block of text with each line broken into a separate span.
Renders a block of text with each line broken into a separate span and links highlighted.
"""
attr :text, :string, required: true
def break_on_newline(assigns) do
broken_text =
assigns.text
|> String.split("\n", trim: false)
|> Enum.intersperse(Phoenix.HTML.Tag.tag(:span, class: "inline-block mt-2"))
def render_description(assigns) do
formatted_text =
Regex.split(~r{https?://\S+}, assigns.text, include_captures: true)
|> Enum.map(fn
"http" <> _ = url ->
Phoenix.HTML.Tag.content_tag(:a, url, class: "text-blue-500 hover:text-blue-300", href: url, target: "_blank")
assigns = Map.put(assigns, :text, broken_text)
text ->
text
|> String.split("\n", trim: false)
|> Enum.intersperse(Phoenix.HTML.Tag.tag(:span, class: "inline-block mt-2"))
end)
assigns = Map.put(assigns, :text, formatted_text)
~H"""
<span><%= @text %></span>
+1 -1
View File
@@ -64,7 +64,7 @@ defmodule PinchflatWeb.Layouts do
<.icon name={@icon} /> <%= @text %>
</span>
<span class="text-bodydark2">
<.icon name="hero-chevron-up" x-bind:class="{ 'rotate-180': selected }" />
<.icon name="hero-chevron-down" x-bind:class="{ 'rotate-180': selected }" />
</span>
</span>
@@ -3,7 +3,7 @@ defmodule Pinchflat.UpgradeButtonLive do
def render(assigns) do
~H"""
<form id="upgradeForm" phx-change="check_matching_text" phx-hook="supressEnterSubmission">
<form id="upgradeForm" phx-change="check_matching_text" phx-hook="supress-enter-submission">
<.input type="text" name="unlock-pro-textbox" value="" />
</form>
+1 -8
View File
@@ -1,14 +1,7 @@
defmodule PinchflatWeb.ErrorHTML do
use PinchflatWeb, :html
# If you want to customize your error pages,
# uncomment the embed_templates/1 call below
# and add pages to the error directory:
#
# * lib/pinchflat_web/controllers/error_html/404.html.heex
# * lib/pinchflat_web/controllers/error_html/500.html.heex
#
# embed_templates "error_html/*"
embed_templates "error_html/*"
# The default is to render a plain text page based on
# the template name. For example, "404.html" becomes
@@ -0,0 +1,3 @@
<section>
<h2 class="text-title-md2 font-bold text-white">404 (not found)</h2>
</section>
@@ -0,0 +1,29 @@
<section>
<h2 class="text-title-md2 font-bold text-white">Internal Server Error</h2>
<p class="text-body-md text-white mt-2">
This shouldn't happen! Please make a
<.inline_link href="https://github.com/kieraneglin/pinchflat/issues/new/choose">GitHub issue</.inline_link>
with the following information:
</p>
<ul class="list-disc ml-8 mb-8">
<li>What you were doing when you saw this page</li>
<li>
Your system details and logs from
<.inline_link href={~p"/app_info"}>app info</.inline_link>
</li>
<li>All the information in the textarea below (use select all + copy)</li>
</ul>
<textarea class="w-full min-h-96 font-mono inline-block rounded-lg" readonly>
**Status**:
`<%= if Map.has_key?(assigns, :status), do: @status, else: "" %>`
**Reason**:
`<%= if Map.has_key?(assigns, :reason), do: inspect(@reason), else: "" %>`
**Stacktrace**:
```
<%= if Map.has_key?(assigns, :stack), do: Exception.format_stacktrace(@stack), else: "" %>
```
</textarea>
</section>
@@ -2,7 +2,7 @@
<:option>
<span x-data="{ copied: false }" x-on:click={~s"
copyWithCallbacks(
`#{Jason.Formatter.pretty_print(Phoenix.json_library().encode!(@media_item))}`,
String.raw`#{Jason.Formatter.pretty_print(Phoenix.json_library().encode!(@media_item))}`,
() => copied = true,
() => copied = false
)
@@ -1,13 +1,13 @@
<%= if media_type(@media_item) == :video do %>
<video controls class="max-h-96 w-full">
<source src={~p"/media/#{@media_item.uuid}/stream"} type="video/mp4" />
<video controls class="max-h-128 w-full">
<source src={~p"/media/#{@media_item.uuid}/stream?v=#{DateTime.to_unix(@media_item.updated_at)}"} type="video/mp4" />
Your browser does not support the video element.
</video>
<% end %>
<%= if media_type(@media_item) == :audio do %>
<audio controls class="w-full">
<source src={~p"/media/#{@media_item.uuid}/stream"} type="audio/mpeg" />
<source src={~p"/media/#{@media_item.uuid}/stream?v=#{DateTime.to_unix(@media_item.updated_at)}"} type="audio/mpeg" />
Your browser does not support the audio element.
</audio>
<% end %>
@@ -26,11 +26,12 @@
<:tab title="Media" id="media">
<div class="flex flex-col gap-10 dark:text-white">
<%= if media_file_exists?(@media_item) do %>
<section class="grid grid-cols-1 xl:grid-cols-2 xl:gap-6 mt-6">
<section class="grid grid-cols-1 xl:gap-6 mt-6">
<div>
<.media_preview media_item={@media_item} />
</div>
<aside class="mt-4 xl:mt-0">
<aside class="mt-4">
<h2 class="text-xl mb-2"><%= @media_item.title %></h2>
<div>Uploaded: <%= DateTime.to_date(@media_item.uploaded_at) %></div>
<div>
<span :if={URI.parse(@media_item.original_url).scheme =~ "http"}>
@@ -38,13 +39,16 @@
<span class="mx-2">or</span>
</span>
<span>
<.subtle_link href={~p"/media/#{@media_item.uuid}/stream"} target="_blank">
<.subtle_link
href={~p"/media/#{@media_item.uuid}/stream?v=#{DateTime.to_unix(@media_item.updated_at)}"}
target="_blank"
>
Open Local Stream
</.subtle_link>
</span>
</div>
<div :if={@media_item.description} class="mt-4 text-bodydark">
<.break_on_newline text={@media_item.description} />
<.render_description text={@media_item.description} />
</div>
</aside>
</section>
@@ -70,7 +74,7 @@
<%= task.job.state %>
</:col>
<:col :let={task} label="Scheduled At">
<%= Calendar.strftime(task.job.scheduled_at, "%y-%m-%d %I:%M:%S %p %Z") %>
<.datetime_in_zone datetime={task.job.scheduled_at} />
</:col>
</.table>
<% else %>
@@ -1,24 +1,51 @@
defmodule PinchflatWeb.MediaProfiles.MediaProfileController do
use PinchflatWeb, :controller
use Pinchflat.Sources.SourcesQuery
use Pinchflat.Profiles.ProfilesQuery
alias Pinchflat.Repo
alias Pinchflat.Profiles
alias Pinchflat.Sources.Source
alias Pinchflat.Profiles.MediaProfile
alias Pinchflat.Profiles.MediaProfileDeletionWorker
def index(conn, _params) do
media_profiles =
MediaProfile
|> order_by(asc: :name)
|> Repo.all()
media_profiles_query =
from mp in MediaProfile,
as: :media_profile,
where: is_nil(mp.marked_for_deletion_at),
order_by: [asc: mp.name],
select: map(mp, ^MediaProfile.__schema__(:fields)),
select_merge: %{
source_count:
subquery(
from s in Source,
where: s.media_profile_id == parent_as(:media_profile).id,
select: count(s.id)
)
}
render(conn, :index, media_profiles: media_profiles)
render(conn, :index, media_profiles: Repo.all(media_profiles_query))
end
def new(conn, _params) do
changeset = Profiles.change_media_profile(%MediaProfile{})
def new(conn, params) do
# Preload an existing media profile for faster creation
cs_struct =
case to_string(params["template_id"]) do
"" -> %MediaProfile{}
template_id -> Repo.get(MediaProfile, template_id) || %MediaProfile{}
end
render(conn, :new, changeset: changeset, layout: get_onboarding_layout())
render(conn, :new,
layout: get_onboarding_layout(),
changeset:
Profiles.change_media_profile(%MediaProfile{
cs_struct
| id: nil,
name: nil,
marked_for_deletion_at: nil
})
)
end
def create(conn, %{"media_profile" => media_profile_params}) do
@@ -70,19 +97,15 @@ defmodule PinchflatWeb.MediaProfiles.MediaProfileController do
end
def delete(conn, %{"id" => id} = params) do
delete_files = Map.get(params, "delete_files", false)
# This awkward comparison converts the string to a boolean
delete_files = Map.get(params, "delete_files", "") == "true"
media_profile = Profiles.get_media_profile!(id)
{:ok, _media_profile} = Profiles.delete_media_profile(media_profile, delete_files: delete_files)
flash_message =
if delete_files do
"Media profile, its sources, and its files deleted successfully."
else
"Media profile and its sources deleted successfully. Files were not deleted."
end
{:ok, _} = Profiles.update_media_profile(media_profile, %{marked_for_deletion_at: DateTime.utc_now()})
MediaProfileDeletionWorker.kickoff(media_profile, %{delete_files: delete_files})
conn
|> put_flash(:info, flash_message)
|> put_flash(:info, "Media Profile deletion started. This may take a while to complete.")
|> redirect(to: ~p"/media_profiles")
end
@@ -10,6 +10,7 @@ defmodule PinchflatWeb.MediaProfiles.MediaProfileHTML do
"""
attr :changeset, Ecto.Changeset, required: true
attr :action, :string, required: true
attr :method, :string, required: true
def media_profile_form(assigns)
@@ -2,7 +2,7 @@
<:option>
<span x-data="{ copied: false }" x-on:click={~s"
copyWithCallbacks(
`#{Jason.Formatter.pretty_print(Phoenix.json_library().encode!(@media_profile))}`,
String.raw`#{Jason.Formatter.pretty_print(Phoenix.json_library().encode!(@media_profile))}`,
() => copied = true,
() => copied = false
)
@@ -11,6 +11,11 @@
<span x-show="copied" x-transition.duration.150ms><.icon name="hero-check" class="ml-2 h-4 w-4" /></span>
</span>
</:option>
<:option>
<.link href={~p"/media_profiles/new?template_id=#{@media_profile}"} method="get">
Use as Template
</.link>
</:option>
<:option>
<div class="h-px w-full bg-bodydark2"></div>
</:option>
@@ -10,7 +10,7 @@
<div class="rounded-sm border border-stroke bg-white px-5 pb-2.5 pt-6 shadow-default dark:border-strokedark dark:bg-boxdark sm:px-7.5 xl:pb-1">
<div class="max-w-full">
<div class="flex flex-col gap-10">
<.media_profile_form changeset={@changeset} action={~p"/media_profiles/#{@media_profile}"} />
<.media_profile_form changeset={@changeset} action={~p"/media_profiles/#{@media_profile}"} method="patch" />
</div>
</div>
</div>
@@ -10,7 +10,6 @@
</.link>
</nav>
</div>
<div class="rounded-sm border border-stroke bg-white shadow-default dark:border-strokedark dark:bg-boxdark">
<div class="max-w-full overflow-x-auto">
<div class="flex flex-col gap-10 min-w-max">
@@ -23,6 +22,11 @@
<:col :let={media_profile} label="Preferred Resolution">
<%= media_profile.preferred_resolution %>
</:col>
<:col :let={media_profile} label="Sources">
<.subtle_link href={~p"/media_profiles/#{media_profile.id}/#tab-sources"}>
<.localized_number number={media_profile.source_count} />
</.subtle_link>
</:col>
<:col :let={media_profile} label="" class="flex justify-end">
<.icon_link href={~p"/media_profiles/#{media_profile.id}/edit"} icon="hero-pencil-square" class="mr-4" />
</:col>
@@ -1,4 +1,11 @@
<.simple_form :let={f} for={@changeset} action={@action}>
<.simple_form
:let={f}
for={@changeset}
action={@action}
method={@method}
x-data="{ advancedMode: !!JSON.parse(localStorage.getItem('advancedMode')) }"
x-init="$watch('advancedMode', value => localStorage.setItem('advancedMode', JSON.stringify(value)))"
>
<.error :if={@changeset.action}>
Oops, something went wrong! Please check the errors below.
</.error>
@@ -30,10 +37,14 @@
</.input>
</section>
<h3 class="mt-8 text-2xl text-black dark:text-white">
General Options
</h3>
<section class="flex justify-between items-center mt-8">
<h3 class="text-2xl text-white">
General Options
</h3>
<span class="cursor-pointer hover:underline" x-on:click="advancedMode = !advancedMode">
Editing Mode: <span x-text="advancedMode ? 'Advanced' : 'Standard'"></span>
</span>
</section>
<section x-data="{
presets: {
default: 'Default',
@@ -114,6 +125,16 @@
/>
</section>
<section x-show="advancedMode">
<.input
field={f[:audio_track]}
placeholder="de"
type="text"
label="Audio Track Language"
help="Only works if there are multiple audio tracks. Use either a language code, 'original' for the original audio track, or 'default' for YouTube's preference. Or just leave it blank"
/>
</section>
<h3 class="mt-10 text-2xl text-black dark:text-white">
Thumbnail Options
</h3>
@@ -203,6 +224,16 @@
/>
</section>
<section x-show="advancedMode">
<.input
field={f[:media_container]}
type="text"
label="Media Container"
placeholder="mp4"
help="Don't change this if you're going to consume media via Plex. Leave blank for default"
/>
</section>
<section x-data="{ presets: { default: null, media_center: 1, audio: null, archiving: 1 } }">
<.input
field={f[:redownload_delay_days]}
@@ -8,7 +8,7 @@
<div class="rounded-sm border border-stroke bg-white px-5 pb-2.5 pt-6 shadow-default dark:border-strokedark dark:bg-boxdark sm:px-7.5 xl:pb-1">
<div class="max-w-full">
<div class="flex flex-col gap-10">
<.media_profile_form changeset={@changeset} action={~p"/media_profiles"} />
<.media_profile_form changeset={@changeset} action={~p"/media_profiles"} method="post" />
</div>
</div>
</div>
@@ -1,44 +1,20 @@
defmodule PinchflatWeb.Sources.SourceController do
use PinchflatWeb, :controller
use Pinchflat.Media.MediaQuery
use Pinchflat.Sources.SourcesQuery
alias Pinchflat.Repo
alias Pinchflat.Tasks
alias Pinchflat.Sources
alias Pinchflat.Sources.Source
alias Pinchflat.Media.MediaItem
alias Pinchflat.Profiles.MediaProfile
alias Pinchflat.Media.FileSyncingWorker
alias Pinchflat.Sources.SourceDeletionWorker
alias Pinchflat.Downloading.DownloadingHelpers
alias Pinchflat.SlowIndexing.SlowIndexingHelpers
alias Pinchflat.Metadata.SourceMetadataStorageWorker
def index(conn, _params) do
source_query =
from s in Source,
as: :source,
inner_join: mp in assoc(s, :media_profile),
preload: [media_profile: mp],
order_by: [asc: s.custom_name],
select: map(s, ^Source.__schema__(:fields)),
select_merge: %{
downloaded_count:
subquery(
from m in MediaItem,
where: m.source_id == parent_as(:source).id,
where: ^MediaQuery.downloaded(),
select: count(m.id)
),
pending_count:
subquery(
from m in MediaItem,
join: s in assoc(m, :source),
where: m.source_id == parent_as(:source).id,
where: ^MediaQuery.pending(),
select: count(m.id)
)
}
render(conn, :index, sources: Repo.all(source_query))
render(conn, :index)
end
def new(conn, params) do
@@ -60,10 +36,12 @@ defmodule PinchflatWeb.Sources.SourceController do
| id: nil,
uuid: nil,
custom_name: nil,
description: nil,
collection_name: nil,
collection_id: nil,
collection_type: nil,
original_url: nil
original_url: nil,
marked_for_deletion_at: nil
})
)
end
@@ -124,19 +102,15 @@ defmodule PinchflatWeb.Sources.SourceController do
end
def delete(conn, %{"id" => id} = params) do
delete_files = Map.get(params, "delete_files", false)
# This awkward comparison converts the string to a boolean
delete_files = Map.get(params, "delete_files", "") == "true"
source = Sources.get_source!(id)
{:ok, _source} = Sources.delete_source(source, delete_files: delete_files)
flash_message =
if delete_files do
"Source and files deleted successfully."
else
"Source deleted successfully. Files were not deleted."
end
{:ok, _} = Sources.update_source(source, %{marked_for_deletion_at: DateTime.utc_now()})
SourceDeletionWorker.kickoff(source, %{delete_files: delete_files})
conn
|> put_flash(:info, flash_message)
|> put_flash(:info, "Source deletion started. This may take a while to complete.")
|> redirect(to: ~p"/sources")
end
@@ -176,6 +150,15 @@ defmodule PinchflatWeb.Sources.SourceController do
)
end
def sync_files_on_disk(conn, %{"source_id" => id}) do
wrap_forced_action(
conn,
id,
"File sync enqueued.",
&FileSyncingWorker.kickoff_with_task/1
)
end
defp wrap_forced_action(conn, source_id, message, fun) do
source = Sources.get_source!(source_id)
fun.(source)
@@ -17,7 +17,7 @@
<:option>
<span x-data="{ copied: false }" x-on:click={~s"
copyWithCallbacks(
`#{Jason.Formatter.pretty_print(Phoenix.json_library().encode!(@source))}`,
String.raw`#{Jason.Formatter.pretty_print(Phoenix.json_library().encode!(@source))}`,
() => copied = true,
() => copied = false
)
@@ -38,7 +38,7 @@
<.link
href={~p"/sources/#{@source}/force_download_pending"}
method="post"
data-confirm="Are you sure you want to force a download of all *pending* media items? This isn't normally needed."
data-confirm="Are you sure you want to force a download of all pending media items? This isn't normally needed."
>
Download Pending
</.link>
@@ -47,7 +47,7 @@
<.link
href={~p"/sources/#{@source}/force_redownload"}
method="post"
data-confirm="Are you sure you want to re-download all currently downloaded media items? This isn't normally needed and won't change anything if the files already exist."
data-confirm="Are you sure you want to re-download all currently downloaded media items? This doesn't upgrade your media, but will download any missing files if your settings have changed. This isn't normally needed."
>
Redownload Existing
</.link>
@@ -70,6 +70,15 @@
Refresh Metadata
</.link>
</:option>
<:option>
<.link
href={~p"/sources/#{@source}/sync_files_on_disk"}
method="post"
data-confirm="Are you sure you want to sync files? This will update media items if their files have been deleted. File addition or moves are not detected. This isn't normally needed."
>
Sync Files on Disk
</.link>
</:option>
<:option>
<div class="h-px w-full bg-bodydark2"></div>
</:option>
@@ -12,32 +12,7 @@
<div class="rounded-sm border border-stroke bg-white shadow-default dark:border-strokedark dark:bg-boxdark">
<div class="max-w-full overflow-x-auto">
<div class="flex flex-col gap-10 min-w-max">
<.table rows={@sources} table_class="text-black dark:text-white">
<:col :let={source} label="Name">
<.subtle_link href={~p"/sources/#{source.id}"}>
<%= StringUtils.truncate(source.custom_name || source.collection_name, 35) %>
</.subtle_link>
</:col>
<:col :let={source} label="Type"><%= source.collection_type %></:col>
<:col :let={source} label="Pending"><.localized_number number={source.pending_count} /></:col>
<:col :let={source} label="Downloaded"><.localized_number number={source.downloaded_count} /></:col>
<:col :let={source} label="Retention">
<%= if source.retention_period_days && source.retention_period_days > 0 do %>
<.localized_number number={source.retention_period_days} />
<.pluralize count={source.retention_period_days} word="day" />
<% else %>
<span class="text-lg">∞</span>
<% end %>
</:col>
<:col :let={source} label="Media Profile">
<.subtle_link href={~p"/media_profiles/#{source.media_profile_id}"}>
<%= source.media_profile.name %>
</.subtle_link>
</:col>
<:col :let={source} label="" class="flex place-content-evenly">
<.icon_link href={~p"/sources/#{source.id}/edit"} icon="hero-pencil-square" class="mx-1" />
</:col>
</.table>
<%= live_render(@conn, PinchflatWeb.Sources.IndexTableLive) %>
</div>
</div>
</div>
@@ -0,0 +1,103 @@
defmodule PinchflatWeb.Sources.IndexTableLive do
use PinchflatWeb, :live_view
use Pinchflat.Media.MediaQuery
use Pinchflat.Sources.SourcesQuery
alias Pinchflat.Repo
alias Pinchflat.Sources
alias Pinchflat.Sources.Source
alias Pinchflat.Media.MediaItem
def render(assigns) do
~H"""
<.table rows={@sources} table_class="text-white">
<:col :let={source} label="Name">
<.subtle_link href={~p"/sources/#{source.id}"}>
<%= StringUtils.truncate(source.custom_name || source.collection_name, 35) %>
</.subtle_link>
</:col>
<:col :let={source} label="Pending">
<.subtle_link href={~p"/sources/#{source.id}/#tab-pending"}>
<.localized_number number={source.pending_count} />
</.subtle_link>
</:col>
<:col :let={source} label="Downloaded">
<.subtle_link href={~p"/sources/#{source.id}/#tab-downloaded"}>
<.localized_number number={source.downloaded_count} />
</.subtle_link>
</:col>
<:col :let={source} label="Retention">
<%= if source.retention_period_days && source.retention_period_days > 0 do %>
<.localized_number number={source.retention_period_days} />
<.pluralize count={source.retention_period_days} word="day" />
<% else %>
<span class="text-lg">∞</span>
<% end %>
</:col>
<:col :let={source} label="Media Profile">
<.subtle_link href={~p"/media_profiles/#{source.media_profile_id}"}>
<%= source.media_profile.name %>
</.subtle_link>
</:col>
<:col :let={source} label="Enabled?">
<.input
name={"source[#{source.id}][enabled]"}
value={source.enabled}
id={"source_#{source.id}_enabled"}
phx-hook="formless-input"
data-subscribe="change"
data-event-name="toggle_enabled"
data-identifier={source.id}
type="toggle"
/>
</:col>
<:col :let={source} label="" class="flex place-content-evenly">
<.icon_link href={~p"/sources/#{source.id}/edit"} icon="hero-pencil-square" class="mx-1" />
</:col>
</.table>
"""
end
def mount(_params, _session, socket) do
{:ok, assign(socket, %{sources: get_sources()})}
end
def handle_event("formless-input", %{"event" => "toggle_enabled"} = params, socket) do
source = Sources.get_source!(params["id"])
should_enable = params["value"] == "true"
{:ok, _} = Sources.update_source(source, %{enabled: should_enable})
{:noreply, assign(socket, %{sources: get_sources()})}
end
defp get_sources do
query =
from s in Source,
as: :source,
inner_join: mp in assoc(s, :media_profile),
where: is_nil(s.marked_for_deletion_at) and is_nil(mp.marked_for_deletion_at),
preload: [media_profile: mp],
order_by: [asc: s.custom_name],
select: map(s, ^Source.__schema__(:fields)),
select_merge: %{
downloaded_count:
subquery(
from m in MediaItem,
where: m.source_id == parent_as(:source).id,
where: ^MediaQuery.downloaded(),
select: count(m.id)
),
pending_count:
subquery(
from m in MediaItem,
join: s in assoc(m, :source),
where: m.source_id == parent_as(:source).id,
where: ^MediaQuery.pending(),
select: count(m.id)
)
}
Repo.all(query)
end
end
@@ -1,4 +1,4 @@
defmodule Pinchflat.Sources.MediaItemTableLive do
defmodule PinchflatWeb.Sources.MediaItemTableLive do
use PinchflatWeb, :live_view
use Pinchflat.Media.MediaQuery
@@ -39,21 +39,21 @@
<:tab title="Pending" id="pending">
<%= live_render(
@conn,
Pinchflat.Sources.MediaItemTableLive,
PinchflatWeb.Sources.MediaItemTableLive,
session: %{"source_id" => @source.id, "media_state" => "pending"}
) %>
</:tab>
<:tab title="Downloaded" id="downloaded">
<%= live_render(
@conn,
Pinchflat.Sources.MediaItemTableLive,
PinchflatWeb.Sources.MediaItemTableLive,
session: %{"source_id" => @source.id, "media_state" => "downloaded"}
) %>
</:tab>
<:tab title="Other" id="other">
<%= live_render(
@conn,
Pinchflat.Sources.MediaItemTableLive,
PinchflatWeb.Sources.MediaItemTableLive,
session: %{"source_id" => @source.id, "media_state" => "other"}
) %>
</:tab>
@@ -67,7 +67,7 @@
<%= task.job.state %>
</:col>
<:col :let={task} label="Scheduled At">
<%= Calendar.strftime(task.job.scheduled_at, "%y-%m-%d %I:%M:%S %p %Z") %>
<.datetime_in_zone datetime={task.job.scheduled_at} />
</:col>
</.table>
<% else %>
@@ -12,7 +12,7 @@
<section x-data="{ mediaProfileId: null }">
<section class="flex justify-between items-center mt-4">
<h3 class=" text-2xl text-black dark:text-white">
<h3 class="text-2xl text-white">
General Options
</h3>
<span class="cursor-pointer hover:underline" x-on:click="advancedMode = !advancedMode">
@@ -86,6 +86,31 @@
help="Unchecking still indexes media but it won't be downloaded until you enable this option"
/>
<.input
field={f[:use_cookies]}
type="toggle"
label="Use Cookies for Downloading"
help="Uses your YouTube cookies for this source (if configured). Used for downloading private playlists and videos. See docs for important details"
/>
<section x-show="advancedMode">
<.input
field={f[:min_duration_seconds]}
type="number"
label="Minimum Duration (seconds)"
min="0"
help="Minimum duration of the media to be downloaded. Can be blank"
/>
<.input
field={f[:max_duration_seconds]}
type="number"
label="Maximum Duration (seconds)"
min="0"
help="Maximum duration of the media to be downloaded. Can be blank"
/>
</section>
<.input
field={f[:download_cutoff_date]}
type="text"
@@ -94,7 +119,7 @@
maxlength="10"
pattern="((?:19|20)[0-9][0-9])-(0[1-9]|1[012])-(0[1-9]|[12][0-9]|3[01])"
title="YYYY-MM-DD"
help="Only download media uploaded after this date. Leave blank to download all media. Must be in YYYY-MM-DD format"
help="Only download media uploaded after this date. Leave blank to download all media. Must be in YYYY-MM-DD format. Old media may be deleted or downloaded if you change this date"
/>
<.input
+1
View File
@@ -40,6 +40,7 @@ defmodule PinchflatWeb.Router do
post "/force_redownload", Sources.SourceController, :force_redownload
post "/force_index", Sources.SourceController, :force_index
post "/force_metadata_refresh", Sources.SourceController, :force_metadata_refresh
post "/sync_files_on_disk", Sources.SourceController, :sync_files_on_disk
resources "/media", MediaItems.MediaItemController, only: [:show, :edit, :update, :delete] do
post "/force_download", MediaItems.MediaItemController, :force_download
+7 -6
View File
@@ -4,7 +4,7 @@ defmodule Pinchflat.MixProject do
def project do
[
app: :pinchflat,
version: "2024.7.19-1",
version: "2024.11.27",
elixir: "~> 1.17",
elixirc_paths: elixirc_paths(Mix.env()),
start_permanent: Mix.env() == :prod,
@@ -46,10 +46,10 @@ defmodule Pinchflat.MixProject do
# Type `mix help deps` for examples and options.
defp deps do
[
{:phoenix, "~> 1.7.10"},
{:phoenix, "~> 1.7.14"},
{:phoenix_ecto, "~> 4.4"},
{:ecto, "~> 3.11.2"},
{:ecto_sql, "~> 3.10"},
{:ecto, "~> 3.12.3"},
{:ecto_sql, "~> 3.12"},
{:ecto_sqlite3, ">= 0.0.0"},
{:ecto_sqlite3_extras, "~> 1.2.0"},
{:phoenix_html, "~> 3.3"},
@@ -67,7 +67,7 @@ defmodule Pinchflat.MixProject do
{:jason, "~> 1.2"},
{:dns_cluster, "~> 0.1.1"},
{:plug_cowboy, "~> 2.5"},
{:oban, "~> 2.16"},
{:oban, "~> 2.17"},
{:nimble_parsec, "~> 1.4"},
{:timex, "~> 3.0"},
{:mox, "~> 1.0", only: :test},
@@ -103,7 +103,8 @@ defmodule Pinchflat.MixProject do
"ecto.rollback": [
"ecto.rollback",
~s(cmd [ -z "$MIX_ENV" ] && yarn run create-erd || echo "No ERD generated")
]
],
"version.bump": "cmd ./tooling/version_bump.sh"
]
end
end
+21 -21
View File
@@ -1,26 +1,26 @@
%{
"bunt": {:hex, :bunt, "1.0.0", "081c2c665f086849e6d57900292b3a161727ab40431219529f13c4ddcf3e7a44", [:mix], [], "hexpm", "dc5f86aa08a5f6fa6b8096f0735c4e76d54ae5c9fa2c143e5a1fc7c1cd9bb6b5"},
"castore": {:hex, :castore, "1.0.5", "9eeebb394cc9a0f3ae56b813459f990abb0a3dedee1be6b27fdb50301930502f", [:mix], [], "hexpm", "8d7c597c3e4a64c395980882d4bca3cebb8d74197c590dc272cfd3b6a6310578"},
"cc_precompiler": {:hex, :cc_precompiler, "0.1.9", "e8d3364f310da6ce6463c3dd20cf90ae7bbecbf6c5203b98bf9b48035592649b", [:mix], [{:elixir_make, "~> 0.7", [hex: :elixir_make, repo: "hexpm", optional: false]}], "hexpm", "9dcab3d0f3038621f1601f13539e7a9ee99843862e66ad62827b0c42b2f58a54"},
"castore": {:hex, :castore, "1.0.8", "dedcf20ea746694647f883590b82d9e96014057aff1d44d03ec90f36a5c0dc6e", [:mix], [], "hexpm", "0b2b66d2ee742cb1d9cb8c8be3b43c3a70ee8651f37b75a8b982e036752983f1"},
"cc_precompiler": {:hex, :cc_precompiler, "0.1.10", "47c9c08d8869cf09b41da36538f62bc1abd3e19e41701c2cea2675b53c704258", [:mix], [{:elixir_make, "~> 0.7", [hex: :elixir_make, repo: "hexpm", optional: false]}], "hexpm", "f6e046254e53cd6b41c6bacd70ae728011aa82b2742a80d6e2214855c6e06b22"},
"certifi": {:hex, :certifi, "2.12.0", "2d1cca2ec95f59643862af91f001478c9863c2ac9cb6e2f89780bfd8de987329", [:rebar3], [], "hexpm", "ee68d85df22e554040cdb4be100f33873ac6051387baf6a8f6ce82272340ff1c"},
"combine": {:hex, :combine, "0.10.0", "eff8224eeb56498a2af13011d142c5e7997a80c8f5b97c499f84c841032e429f", [:mix], [], "hexpm", "1b1dbc1790073076580d0d1d64e42eae2366583e7aecd455d1215b0d16f2451b"},
"cowboy": {:hex, :cowboy, "2.10.0", "ff9ffeff91dae4ae270dd975642997afe2a1179d94b1887863e43f681a203e26", [:make, :rebar3], [{:cowlib, "2.12.1", [hex: :cowlib, repo: "hexpm", optional: false]}, {:ranch, "1.8.0", [hex: :ranch, repo: "hexpm", optional: false]}], "hexpm", "3afdccb7183cc6f143cb14d3cf51fa00e53db9ec80cdcd525482f5e99bc41d6b"},
"cowboy": {:hex, :cowboy, "2.12.0", "f276d521a1ff88b2b9b4c54d0e753da6c66dd7be6c9fca3d9418b561828a3731", [:make, :rebar3], [{:cowlib, "2.13.0", [hex: :cowlib, repo: "hexpm", optional: false]}, {:ranch, "1.8.0", [hex: :ranch, repo: "hexpm", optional: false]}], "hexpm", "8a7abe6d183372ceb21caa2709bec928ab2b72e18a3911aa1771639bef82651e"},
"cowboy_telemetry": {:hex, :cowboy_telemetry, "0.4.0", "f239f68b588efa7707abce16a84d0d2acf3a0f50571f8bb7f56a15865aae820c", [:rebar3], [{:cowboy, "~> 2.7", [hex: :cowboy, repo: "hexpm", optional: false]}, {:telemetry, "~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "7d98bac1ee4565d31b62d59f8823dfd8356a169e7fcbb83831b8a5397404c9de"},
"cowlib": {:hex, :cowlib, "2.12.1", "a9fa9a625f1d2025fe6b462cb865881329b5caff8f1854d1cbc9f9533f00e1e1", [:make, :rebar3], [], "hexpm", "163b73f6367a7341b33c794c4e88e7dbfe6498ac42dcd69ef44c5bc5507c8db0"},
"cowlib": {:hex, :cowlib, "2.13.0", "db8f7505d8332d98ef50a3ef34b34c1afddec7506e4ee4dd4a3a266285d282ca", [:make, :rebar3], [], "hexpm", "e1e1284dc3fc030a64b1ad0d8382ae7e99da46c3246b815318a4b848873800a4"},
"credo": {:hex, :credo, "1.7.7", "771445037228f763f9b2afd612b6aa2fd8e28432a95dbbc60d8e03ce71ba4446", [:mix], [{:bunt, "~> 0.2.1 or ~> 1.0", [hex: :bunt, repo: "hexpm", optional: false]}, {:file_system, "~> 0.2 or ~> 1.0", [hex: :file_system, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}], "hexpm", "8bc87496c9aaacdc3f90f01b7b0582467b69b4bd2441fe8aae3109d843cc2f2e"},
"credo_naming": {:hex, :credo_naming, "2.1.0", "d44ad58890d4db552e141ce64756a74ac1573665af766d1ac64931aa90d47744", [:make, :mix], [{:credo, "~> 1.6", [hex: :credo, repo: "hexpm", optional: false]}], "hexpm", "830e23b3fba972e2fccec49c0c089fe78c1e64bc16782a2682d78082351a2909"},
"db_connection": {:hex, :db_connection, "2.6.0", "77d835c472b5b67fc4f29556dee74bf511bbafecdcaf98c27d27fa5918152086", [:mix], [{:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "c2f992d15725e721ec7fbc1189d4ecdb8afef76648c746a8e1cad35e3b8a35f3"},
"db_connection": {:hex, :db_connection, "2.7.0", "b99faa9291bb09892c7da373bb82cba59aefa9b36300f6145c5f201c7adf48ec", [:mix], [{:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "dcf08f31b2701f857dfc787fbad78223d61a32204f217f15e881dd93e4bdd3ff"},
"decimal": {:hex, :decimal, "2.1.1", "5611dca5d4b2c3dd497dec8f68751f1f1a54755e8ed2a966c2633cf885973ad6", [:mix], [], "hexpm", "53cfe5f497ed0e7771ae1a475575603d77425099ba5faef9394932b35020ffcc"},
"dns_cluster": {:hex, :dns_cluster, "0.1.2", "3eb5be824c7888dadf9781018e1a5f1d3d1113b333c50bce90fb1b83df1015f2", [:mix], [], "hexpm", "7494272040f847637bbdb01bcdf4b871e82daf09b813e7d3cb3b84f112c6f2f8"},
"ecto": {:hex, :ecto, "3.11.2", "e1d26be989db350a633667c5cda9c3d115ae779b66da567c68c80cfb26a8c9ee", [:mix], [{:decimal, "~> 2.0", [hex: :decimal, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: true]}, {:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "3c38bca2c6f8d8023f2145326cc8a80100c3ffe4dcbd9842ff867f7fc6156c65"},
"ecto_sql": {:hex, :ecto_sql, "3.11.1", "e9abf28ae27ef3916b43545f9578b4750956ccea444853606472089e7d169470", [:mix], [{:db_connection, "~> 2.4.1 or ~> 2.5", [hex: :db_connection, repo: "hexpm", optional: false]}, {:ecto, "~> 3.11.0", [hex: :ecto, repo: "hexpm", optional: false]}, {:myxql, "~> 0.6.0", [hex: :myxql, repo: "hexpm", optional: true]}, {:postgrex, "~> 0.16.0 or ~> 0.17.0 or ~> 1.0", [hex: :postgrex, repo: "hexpm", optional: true]}, {:tds, "~> 2.1.1 or ~> 2.2", [hex: :tds, repo: "hexpm", optional: true]}, {:telemetry, "~> 0.4.0 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "ce14063ab3514424276e7e360108ad6c2308f6d88164a076aac8a387e1fea634"},
"ecto_sqlite3": {:hex, :ecto_sqlite3, "0.15.1", "40f2fbd9e246455f8c42e7e0a77009ef806caa1b3ce6f717b2a0a80e8432fcfd", [:mix], [{:decimal, "~> 1.6 or ~> 2.0", [hex: :decimal, repo: "hexpm", optional: false]}, {:ecto, "~> 3.11", [hex: :ecto, repo: "hexpm", optional: false]}, {:ecto_sql, "~> 3.11", [hex: :ecto_sql, repo: "hexpm", optional: false]}, {:exqlite, "~> 0.19", [hex: :exqlite, repo: "hexpm", optional: false]}], "hexpm", "28b16e177123c688948357176662bf9ff9084daddf950ef5b6baf3ee93707064"},
"ecto": {:hex, :ecto, "3.12.3", "1a9111560731f6c3606924c81c870a68a34c819f6d4f03822f370ea31a582208", [:mix], [{:decimal, "~> 2.0", [hex: :decimal, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: true]}, {:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "9efd91506ae722f95e48dc49e70d0cb632ede3b7a23896252a60a14ac6d59165"},
"ecto_sql": {:hex, :ecto_sql, "3.12.0", "73cea17edfa54bde76ee8561b30d29ea08f630959685006d9c6e7d1e59113b7d", [:mix], [{:db_connection, "~> 2.4.1 or ~> 2.5", [hex: :db_connection, repo: "hexpm", optional: false]}, {:ecto, "~> 3.12", [hex: :ecto, repo: "hexpm", optional: false]}, {:myxql, "~> 0.7", [hex: :myxql, repo: "hexpm", optional: true]}, {:postgrex, "~> 0.19 or ~> 1.0", [hex: :postgrex, repo: "hexpm", optional: true]}, {:tds, "~> 2.1.1 or ~> 2.2", [hex: :tds, repo: "hexpm", optional: true]}, {:telemetry, "~> 0.4.0 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "dc9e4d206f274f3947e96142a8fdc5f69a2a6a9abb4649ef5c882323b6d512f0"},
"ecto_sqlite3": {:hex, :ecto_sqlite3, "0.17.2", "200226e057f76c40be55fbac77771eb1a233260ab8ec7283f5da6d9402bde8de", [:mix], [{:decimal, "~> 1.6 or ~> 2.0", [hex: :decimal, repo: "hexpm", optional: false]}, {:ecto, "~> 3.12", [hex: :ecto, repo: "hexpm", optional: false]}, {:ecto_sql, "~> 3.12", [hex: :ecto_sql, repo: "hexpm", optional: false]}, {:exqlite, "~> 0.22", [hex: :exqlite, repo: "hexpm", optional: false]}], "hexpm", "a3838919c5a34c268c28cafab87b910bcda354a9a4e778658da46c149bb2c1da"},
"ecto_sqlite3_extras": {:hex, :ecto_sqlite3_extras, "1.2.2", "36e60b561a11441d15f26c791817999269fb578b985162207ebb08b04ca71e40", [:mix], [{:exqlite, ">= 0.13.2", [hex: :exqlite, repo: "hexpm", optional: false]}, {:table_rex, "~> 4.0", [hex: :table_rex, repo: "hexpm", optional: false]}], "hexpm", "2b66ba7246bb4f7e39e2578acd4a0e4e4be54f60ff52d450a01be95eeb78ff1e"},
"elixir_make": {:hex, :elixir_make, "0.7.8", "505026f266552ee5aabca0b9f9c229cbb496c689537c9f922f3eb5431157efc7", [:mix], [{:castore, "~> 0.1 or ~> 1.0", [hex: :castore, repo: "hexpm", optional: true]}, {:certifi, "~> 2.0", [hex: :certifi, repo: "hexpm", optional: true]}], "hexpm", "7a71945b913d37ea89b06966e1342c85cfe549b15e6d6d081e8081c493062c07"},
"elixir_make": {:hex, :elixir_make, "0.8.4", "4960a03ce79081dee8fe119d80ad372c4e7badb84c493cc75983f9d3bc8bde0f", [:mix], [{:castore, "~> 0.1 or ~> 1.0", [hex: :castore, repo: "hexpm", optional: true]}, {:certifi, "~> 2.0", [hex: :certifi, repo: "hexpm", optional: true]}], "hexpm", "6e7f1d619b5f61dfabd0a20aa268e575572b542ac31723293a4c1a567d5ef040"},
"esbuild": {:hex, :esbuild, "0.8.1", "0cbf919f0eccb136d2eeef0df49c4acf55336de864e63594adcea3814f3edf41", [:mix], [{:castore, ">= 0.0.0", [hex: :castore, repo: "hexpm", optional: false]}, {:jason, "~> 1.4", [hex: :jason, repo: "hexpm", optional: false]}], "hexpm", "25fc876a67c13cb0a776e7b5d7974851556baeda2085296c14ab48555ea7560f"},
"ex_check": {:hex, :ex_check, "0.14.0", "d6fbe0bcc51cf38fea276f5bc2af0c9ae0a2bb059f602f8de88709421dae4f0e", [:mix], [], "hexpm", "8a602e98c66e6a4be3a639321f1f545292042f290f91fa942a285888c6868af0"},
"expo": {:hex, :expo, "0.5.1", "249e826a897cac48f591deba863b26c16682b43711dd15ee86b92f25eafd96d9", [:mix], [], "hexpm", "68a4233b0658a3d12ee00d27d37d856b1ba48607e7ce20fd376958d0ba6ce92b"},
"exqlite": {:hex, :exqlite, "0.19.0", "0f3ee29e35bed38552dd0ed59600aa81c78f867f5b5ff0e17d330148e0465483", [:make, :mix], [{:cc_precompiler, "~> 0.1", [hex: :cc_precompiler, repo: "hexpm", optional: false]}, {:db_connection, "~> 2.1", [hex: :db_connection, repo: "hexpm", optional: false]}, {:elixir_make, "~> 0.7", [hex: :elixir_make, repo: "hexpm", optional: false]}, {:table, "~> 0.1.0", [hex: :table, repo: "hexpm", optional: true]}], "hexpm", "55a8fbb0443f03d4a256e3458bd1203eff5037a6624b76460eaaa9080f462b06"},
"exqlite": {:hex, :exqlite, "0.23.0", "6e851c937a033299d0784994c66da24845415072adbc455a337e20087bce9033", [:make, :mix], [{:cc_precompiler, "~> 0.1", [hex: :cc_precompiler, repo: "hexpm", optional: false]}, {:db_connection, "~> 2.1", [hex: :db_connection, repo: "hexpm", optional: false]}, {:elixir_make, "~> 0.8", [hex: :elixir_make, repo: "hexpm", optional: false]}, {:table, "~> 0.1.0", [hex: :table, repo: "hexpm", optional: true]}], "hexpm", "404341cceec5e6466aaed160cf0b58be2019b60af82588c215e1224ebd3ec831"},
"faker": {:hex, :faker, "0.17.0", "671019d0652f63aefd8723b72167ecdb284baf7d47ad3a82a15e9b8a6df5d1fa", [:mix], [], "hexpm", "a7d4ad84a93fd25c5f5303510753789fc2433ff241bf3b4144d3f6f291658a6a"},
"file_system": {:hex, :file_system, "0.2.10", "fb082005a9cd1711c05b5248710f8826b02d7d1784e7c3451f9c1231d4fc162d", [:mix], [], "hexpm", "41195edbfb562a593726eda3b3e8b103a309b733ad25f3d642ba49696bf715dc"},
"finch": {:hex, :finch, "0.17.0", "17d06e1d44d891d20dbd437335eebe844e2426a0cd7e3a3e220b461127c73f70", [:mix], [{:castore, "~> 0.1 or ~> 1.0", [hex: :castore, repo: "hexpm", optional: false]}, {:mime, "~> 1.0 or ~> 2.0", [hex: :mime, repo: "hexpm", optional: false]}, {:mint, "~> 1.3", [hex: :mint, repo: "hexpm", optional: false]}, {:nimble_options, "~> 0.4 or ~> 1.0", [hex: :nimble_options, repo: "hexpm", optional: false]}, {:nimble_pool, "~> 0.2.6 or ~> 1.0", [hex: :nimble_pool, repo: "hexpm", optional: false]}, {:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "8d014a661bb6a437263d4b5abf0bcbd3cf0deb26b1e8596f2a271d22e48934c7"},
@@ -29,40 +29,40 @@
"hackney": {:hex, :hackney, "1.20.1", "8d97aec62ddddd757d128bfd1df6c5861093419f8f7a4223823537bad5d064e2", [:rebar3], [{:certifi, "~> 2.12.0", [hex: :certifi, repo: "hexpm", optional: false]}, {:idna, "~> 6.1.0", [hex: :idna, repo: "hexpm", optional: false]}, {:metrics, "~> 1.0.0", [hex: :metrics, repo: "hexpm", optional: false]}, {:mimerl, "~> 1.1", [hex: :mimerl, repo: "hexpm", optional: false]}, {:parse_trans, "3.4.1", [hex: :parse_trans, repo: "hexpm", optional: false]}, {:ssl_verify_fun, "~> 1.1.0", [hex: :ssl_verify_fun, repo: "hexpm", optional: false]}, {:unicode_util_compat, "~> 0.7.0", [hex: :unicode_util_compat, repo: "hexpm", optional: false]}], "hexpm", "fe9094e5f1a2a2c0a7d10918fee36bfec0ec2a979994cff8cfe8058cd9af38e3"},
"hpax": {:hex, :hpax, "0.1.2", "09a75600d9d8bbd064cdd741f21fc06fc1f4cf3d0fcc335e5aa19be1a7235c84", [:mix], [], "hexpm", "2c87843d5a23f5f16748ebe77969880e29809580efdaccd615cd3bed628a8c13"},
"idna": {:hex, :idna, "6.1.1", "8a63070e9f7d0c62eb9d9fcb360a7de382448200fbbd1b106cc96d3d8099df8d", [:rebar3], [{:unicode_util_compat, "~> 0.7.0", [hex: :unicode_util_compat, repo: "hexpm", optional: false]}], "hexpm", "92376eb7894412ed19ac475e4a86f7b413c1b9fbb5bd16dccd57934157944cea"},
"jason": {:hex, :jason, "1.4.1", "af1504e35f629ddcdd6addb3513c3853991f694921b1b9368b0bd32beb9f1b63", [:mix], [{:decimal, "~> 1.0 or ~> 2.0", [hex: :decimal, repo: "hexpm", optional: true]}], "hexpm", "fbb01ecdfd565b56261302f7e1fcc27c4fb8f32d56eab74db621fc154604a7a1"},
"jason": {:hex, :jason, "1.4.4", "b9226785a9aa77b6857ca22832cffa5d5011a667207eb2a0ad56adb5db443b8a", [:mix], [{:decimal, "~> 1.0 or ~> 2.0", [hex: :decimal, repo: "hexpm", optional: true]}], "hexpm", "c5eb0cab91f094599f94d55bc63409236a8ec69a21a67814529e8d5f6cc90b3b"},
"metrics": {:hex, :metrics, "1.0.1", "25f094dea2cda98213cecc3aeff09e940299d950904393b2a29d191c346a8486", [:rebar3], [], "hexpm", "69b09adddc4f74a40716ae54d140f93beb0fb8978d8636eaded0c31b6f099f16"},
"mime": {:hex, :mime, "2.0.5", "dc34c8efd439abe6ae0343edbb8556f4d63f178594894720607772a041b04b02", [:mix], [], "hexpm", "da0d64a365c45bc9935cc5c8a7fc5e49a0e0f9932a761c55d6c52b142780a05c"},
"mime": {:hex, :mime, "2.0.6", "8f18486773d9b15f95f4f4f1e39b710045fa1de891fada4516559967276e4dc2", [:mix], [], "hexpm", "c9945363a6b26d747389aac3643f8e0e09d30499a138ad64fe8fd1d13d9b153e"},
"mimerl": {:hex, :mimerl, "1.3.0", "d0cd9fc04b9061f82490f6581e0128379830e78535e017f7780f37fea7545726", [:rebar3], [], "hexpm", "a1e15a50d1887217de95f0b9b0793e32853f7c258a5cd227650889b38839fe9d"},
"mint": {:hex, :mint, "1.5.2", "4805e059f96028948870d23d7783613b7e6b0e2fb4e98d720383852a760067fd", [:mix], [{:castore, "~> 0.1.0 or ~> 1.0", [hex: :castore, repo: "hexpm", optional: true]}, {:hpax, "~> 0.1.1", [hex: :hpax, repo: "hexpm", optional: false]}], "hexpm", "d77d9e9ce4eb35941907f1d3df38d8f750c357865353e21d335bdcdf6d892a02"},
"mox": {:hex, :mox, "1.1.0", "0f5e399649ce9ab7602f72e718305c0f9cdc351190f72844599545e4996af73c", [:mix], [], "hexpm", "d44474c50be02d5b72131070281a5d3895c0e7a95c780e90bc0cfe712f633a13"},
"nimble_options": {:hex, :nimble_options, "1.1.0", "3b31a57ede9cb1502071fade751ab0c7b8dbe75a9a4c2b5bbb0943a690b63172", [:mix], [], "hexpm", "8bbbb3941af3ca9acc7835f5655ea062111c9c27bcac53e004460dfd19008a99"},
"nimble_parsec": {:hex, :nimble_parsec, "1.4.0", "51f9b613ea62cfa97b25ccc2c1b4216e81df970acd8e16e8d1bdc58fef21370d", [:mix], [], "hexpm", "9c565862810fb383e9838c1dd2d7d2c437b3d13b267414ba6af33e50d2d1cf28"},
"nimble_pool": {:hex, :nimble_pool, "1.0.0", "5eb82705d138f4dd4423f69ceb19ac667b3b492ae570c9f5c900bb3d2f50a847", [:mix], [], "hexpm", "80be3b882d2d351882256087078e1b1952a28bf98d0a287be87e4a24a710b67a"},
"oban": {:hex, :oban, "2.17.3", "ddfd5710aadcd550d2e174c8d73ce5f1865601418cf54a91775f20443fb832b7", [:mix], [{:ecto_sql, "~> 3.6", [hex: :ecto_sql, repo: "hexpm", optional: false]}, {:ecto_sqlite3, "~> 0.9", [hex: :ecto_sqlite3, repo: "hexpm", optional: true]}, {:jason, "~> 1.1", [hex: :jason, repo: "hexpm", optional: false]}, {:postgrex, "~> 0.16", [hex: :postgrex, repo: "hexpm", optional: true]}, {:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "452eada8bfe0d0fefd0740ab5fa8cf3ef6c375df0b4a3c3805d179022a04738a"},
"oban": {:hex, :oban, "2.18.3", "1608c04f8856c108555c379f2f56bc0759149d35fa9d3b825cb8a6769f8ae926", [:mix], [{:ecto_sql, "~> 3.10", [hex: :ecto_sql, repo: "hexpm", optional: false]}, {:ecto_sqlite3, "~> 0.9", [hex: :ecto_sqlite3, repo: "hexpm", optional: true]}, {:jason, "~> 1.1", [hex: :jason, repo: "hexpm", optional: false]}, {:postgrex, "~> 0.16", [hex: :postgrex, repo: "hexpm", optional: true]}, {:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "36ca6ca84ef6518f9c2c759ea88efd438a3c81d667ba23b02b062a0aa785475e"},
"parse_trans": {:hex, :parse_trans, "3.4.1", "6e6aa8167cb44cc8f39441d05193be6e6f4e7c2946cb2759f015f8c56b76e5ff", [:rebar3], [], "hexpm", "620a406ce75dada827b82e453c19cf06776be266f5a67cff34e1ef2cbb60e49a"},
"phoenix": {:hex, :phoenix, "1.7.10", "02189140a61b2ce85bb633a9b6fd02dff705a5f1596869547aeb2b2b95edd729", [:mix], [{:castore, ">= 0.0.0", [hex: :castore, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: true]}, {:phoenix_pubsub, "~> 2.1", [hex: :phoenix_pubsub, repo: "hexpm", optional: false]}, {:phoenix_template, "~> 1.0", [hex: :phoenix_template, repo: "hexpm", optional: false]}, {:phoenix_view, "~> 2.0", [hex: :phoenix_view, repo: "hexpm", optional: true]}, {:plug, "~> 1.14", [hex: :plug, repo: "hexpm", optional: false]}, {:plug_cowboy, "~> 2.6", [hex: :plug_cowboy, repo: "hexpm", optional: true]}, {:plug_crypto, "~> 1.2 or ~> 2.0", [hex: :plug_crypto, repo: "hexpm", optional: false]}, {:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}, {:websock_adapter, "~> 0.5.3", [hex: :websock_adapter, repo: "hexpm", optional: false]}], "hexpm", "cf784932e010fd736d656d7fead6a584a4498efefe5b8227e9f383bf15bb79d0"},
"phoenix": {:hex, :phoenix, "1.7.14", "a7d0b3f1bc95987044ddada111e77bd7f75646a08518942c72a8440278ae7825", [:mix], [{:castore, ">= 0.0.0", [hex: :castore, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: true]}, {:phoenix_pubsub, "~> 2.1", [hex: :phoenix_pubsub, repo: "hexpm", optional: false]}, {:phoenix_template, "~> 1.0", [hex: :phoenix_template, repo: "hexpm", optional: false]}, {:phoenix_view, "~> 2.0", [hex: :phoenix_view, repo: "hexpm", optional: true]}, {:plug, "~> 1.14", [hex: :plug, repo: "hexpm", optional: false]}, {:plug_cowboy, "~> 2.7", [hex: :plug_cowboy, repo: "hexpm", optional: true]}, {:plug_crypto, "~> 1.2 or ~> 2.0", [hex: :plug_crypto, repo: "hexpm", optional: false]}, {:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}, {:websock_adapter, "~> 0.5.3", [hex: :websock_adapter, repo: "hexpm", optional: false]}], "hexpm", "c7859bc56cc5dfef19ecfc240775dae358cbaa530231118a9e014df392ace61a"},
"phoenix_ecto": {:hex, :phoenix_ecto, "4.4.3", "86e9878f833829c3f66da03d75254c155d91d72a201eb56ae83482328dc7ca93", [:mix], [{:ecto, "~> 3.5", [hex: :ecto, repo: "hexpm", optional: false]}, {:phoenix_html, "~> 2.14.2 or ~> 3.0 or ~> 4.0", [hex: :phoenix_html, repo: "hexpm", optional: true]}, {:plug, "~> 1.9", [hex: :plug, repo: "hexpm", optional: false]}], "hexpm", "d36c401206f3011fefd63d04e8ef626ec8791975d9d107f9a0817d426f61ac07"},
"phoenix_html": {:hex, :phoenix_html, "3.3.3", "380b8fb45912b5638d2f1d925a3771b4516b9a78587249cabe394e0a5d579dc9", [:mix], [{:plug, "~> 1.5", [hex: :plug, repo: "hexpm", optional: true]}], "hexpm", "923ebe6fec6e2e3b3e569dfbdc6560de932cd54b000ada0208b5f45024bdd76c"},
"phoenix_html": {:hex, :phoenix_html, "3.3.4", "42a09fc443bbc1da37e372a5c8e6755d046f22b9b11343bf885067357da21cb3", [:mix], [{:plug, "~> 1.5", [hex: :plug, repo: "hexpm", optional: true]}], "hexpm", "0249d3abec3714aff3415e7ee3d9786cb325be3151e6c4b3021502c585bf53fb"},
"phoenix_live_dashboard": {:hex, :phoenix_live_dashboard, "0.8.3", "7ff51c9b6609470f681fbea20578dede0e548302b0c8bdf338b5a753a4f045bf", [:mix], [{:ecto, "~> 3.6.2 or ~> 3.7", [hex: :ecto, repo: "hexpm", optional: true]}, {:ecto_mysql_extras, "~> 0.5", [hex: :ecto_mysql_extras, repo: "hexpm", optional: true]}, {:ecto_psql_extras, "~> 0.7", [hex: :ecto_psql_extras, repo: "hexpm", optional: true]}, {:ecto_sqlite3_extras, "~> 1.1.7 or ~> 1.2.0", [hex: :ecto_sqlite3_extras, repo: "hexpm", optional: true]}, {:mime, "~> 1.6 or ~> 2.0", [hex: :mime, repo: "hexpm", optional: false]}, {:phoenix_live_view, "~> 0.19 or ~> 1.0", [hex: :phoenix_live_view, repo: "hexpm", optional: false]}, {:telemetry_metrics, "~> 0.6 or ~> 1.0", [hex: :telemetry_metrics, repo: "hexpm", optional: false]}], "hexpm", "f9470a0a8bae4f56430a23d42f977b5a6205fdba6559d76f932b876bfaec652d"},
"phoenix_live_reload": {:hex, :phoenix_live_reload, "1.4.1", "2aff698f5e47369decde4357ba91fc9c37c6487a512b41732818f2204a8ef1d3", [:mix], [{:file_system, "~> 0.2.1 or ~> 0.3", [hex: :file_system, repo: "hexpm", optional: false]}, {:phoenix, "~> 1.4", [hex: :phoenix, repo: "hexpm", optional: false]}], "hexpm", "9bffb834e7ddf08467fe54ae58b5785507aaba6255568ae22b4d46e2bb3615ab"},
"phoenix_live_view": {:hex, :phoenix_live_view, "0.20.3", "8b6406bc0a451f295407d7acff7f234a6314be5bbe0b3f90ed82b07f50049878", [:mix], [{:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: true]}, {:phoenix, "~> 1.6.15 or ~> 1.7.0", [hex: :phoenix, repo: "hexpm", optional: false]}, {:phoenix_html, "~> 3.3 or ~> 4.0", [hex: :phoenix_html, repo: "hexpm", optional: false]}, {:phoenix_template, "~> 1.0", [hex: :phoenix_template, repo: "hexpm", optional: false]}, {:phoenix_view, "~> 2.0", [hex: :phoenix_view, repo: "hexpm", optional: true]}, {:plug, "~> 1.15", [hex: :plug, repo: "hexpm", optional: false]}, {:telemetry, "~> 0.4.2 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "a8e4385e05618b424779f894ed2df97d3c7518b7285fcd11979077ae6226466b"},
"phoenix_pubsub": {:hex, :phoenix_pubsub, "2.1.3", "3168d78ba41835aecad272d5e8cd51aa87a7ac9eb836eabc42f6e57538e3731d", [:mix], [], "hexpm", "bba06bc1dcfd8cb086759f0edc94a8ba2bc8896d5331a1e2c2902bf8e36ee502"},
"phoenix_template": {:hex, :phoenix_template, "1.0.4", "e2092c132f3b5e5b2d49c96695342eb36d0ed514c5b252a77048d5969330d639", [:mix], [{:phoenix_html, "~> 2.14.2 or ~> 3.0 or ~> 4.0", [hex: :phoenix_html, repo: "hexpm", optional: true]}], "hexpm", "2c0c81f0e5c6753faf5cca2f229c9709919aba34fab866d3bc05060c9c444206"},
"plug": {:hex, :plug, "1.15.3", "712976f504418f6dff0a3e554c40d705a9bcf89a7ccef92fc6a5ef8f16a30a97", [:mix], [{:mime, "~> 1.0 or ~> 2.0", [hex: :mime, repo: "hexpm", optional: false]}, {:plug_crypto, "~> 1.1.1 or ~> 1.2 or ~> 2.0", [hex: :plug_crypto, repo: "hexpm", optional: false]}, {:telemetry, "~> 0.4.3 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "cc4365a3c010a56af402e0809208873d113e9c38c401cabd88027ef4f5c01fd2"},
"plug_cowboy": {:hex, :plug_cowboy, "2.6.1", "9a3bbfceeb65eff5f39dab529e5cd79137ac36e913c02067dba3963a26efe9b2", [:mix], [{:cowboy, "~> 2.7", [hex: :cowboy, repo: "hexpm", optional: false]}, {:cowboy_telemetry, "~> 0.3", [hex: :cowboy_telemetry, repo: "hexpm", optional: false]}, {:plug, "~> 1.14", [hex: :plug, repo: "hexpm", optional: false]}], "hexpm", "de36e1a21f451a18b790f37765db198075c25875c64834bcc82d90b309eb6613"},
"plug_crypto": {:hex, :plug_crypto, "2.0.0", "77515cc10af06645abbfb5e6ad7a3e9714f805ae118fa1a70205f80d2d70fe73", [:mix], [], "hexpm", "53695bae57cc4e54566d993eb01074e4d894b65a3766f1c43e2c61a1b0f45ea9"},
"plug": {:hex, :plug, "1.16.1", "40c74619c12f82736d2214557dedec2e9762029b2438d6d175c5074c933edc9d", [:mix], [{:mime, "~> 1.0 or ~> 2.0", [hex: :mime, repo: "hexpm", optional: false]}, {:plug_crypto, "~> 1.1.1 or ~> 1.2 or ~> 2.0", [hex: :plug_crypto, repo: "hexpm", optional: false]}, {:telemetry, "~> 0.4.3 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "a13ff6b9006b03d7e33874945b2755253841b238c34071ed85b0e86057f8cddc"},
"plug_cowboy": {:hex, :plug_cowboy, "2.7.2", "fdadb973799ae691bf9ecad99125b16625b1c6039999da5fe544d99218e662e4", [:mix], [{:cowboy, "~> 2.7", [hex: :cowboy, repo: "hexpm", optional: false]}, {:cowboy_telemetry, "~> 0.3", [hex: :cowboy_telemetry, repo: "hexpm", optional: false]}, {:plug, "~> 1.14", [hex: :plug, repo: "hexpm", optional: false]}], "hexpm", "245d8a11ee2306094840c000e8816f0cbed69a23fc0ac2bcf8d7835ae019bb2f"},
"plug_crypto": {:hex, :plug_crypto, "2.1.0", "f44309c2b06d249c27c8d3f65cfe08158ade08418cf540fd4f72d4d6863abb7b", [:mix], [], "hexpm", "131216a4b030b8f8ce0f26038bc4421ae60e4bb95c5cf5395e1421437824c4fa"},
"ranch": {:hex, :ranch, "1.8.0", "8c7a100a139fd57f17327b6413e4167ac559fbc04ca7448e9be9057311597a1d", [:make, :rebar3], [], "hexpm", "49fbcfd3682fab1f5d109351b61257676da1a2fdbe295904176d5e521a2ddfe5"},
"sobelow": {:hex, :sobelow, "0.13.0", "218afe9075904793f5c64b8837cc356e493d88fddde126a463839351870b8d1e", [:mix], [{:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}], "hexpm", "cd6e9026b85fc35d7529da14f95e85a078d9dd1907a9097b3ba6ac7ebbe34a0d"},
"ssl_verify_fun": {:hex, :ssl_verify_fun, "1.1.7", "354c321cf377240c7b8716899e182ce4890c5938111a1296add3ec74cf1715df", [:make, :mix, :rebar3], [], "hexpm", "fe4c190e8f37401d30167c8c405eda19469f34577987c76dde613e838bbc67f8"},
"swoosh": {:hex, :swoosh, "1.14.4", "94e9dba91f7695a10f49b0172c4a4cb658ef24abef7e8140394521b7f3bbb2d4", [:mix], [{:cowboy, "~> 1.1 or ~> 2.4", [hex: :cowboy, repo: "hexpm", optional: true]}, {:ex_aws, "~> 2.1", [hex: :ex_aws, repo: "hexpm", optional: true]}, {:finch, "~> 0.6", [hex: :finch, repo: "hexpm", optional: true]}, {:gen_smtp, "~> 0.13 or ~> 1.0", [hex: :gen_smtp, repo: "hexpm", optional: true]}, {:hackney, "~> 1.9", [hex: :hackney, repo: "hexpm", optional: true]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}, {:mail, "~> 0.2", [hex: :mail, repo: "hexpm", optional: true]}, {:mime, "~> 1.1 or ~> 2.0", [hex: :mime, repo: "hexpm", optional: false]}, {:plug, "~> 1.9", [hex: :plug, repo: "hexpm", optional: true]}, {:plug_cowboy, ">= 1.0.0", [hex: :plug_cowboy, repo: "hexpm", optional: true]}, {:req, "~> 0.4 or ~> 1.0", [hex: :req, repo: "hexpm", optional: true]}, {:telemetry, "~> 0.4.2 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "081c5a590e4ba85cc89baddf7b2beecf6c13f7f84a958f1cd969290815f0f026"},
"table_rex": {:hex, :table_rex, "4.0.0", "3c613a68ebdc6d4d1e731bc973c233500974ec3993c99fcdabb210407b90959b", [:mix], [], "hexpm", "c35c4d5612ca49ebb0344ea10387da4d2afe278387d4019e4d8111e815df8f55"},
"tailwind": {:hex, :tailwind, "0.2.2", "9e27288b568ede1d88517e8c61259bc214a12d7eed271e102db4c93fcca9b2cd", [:mix], [{:castore, ">= 0.0.0", [hex: :castore, repo: "hexpm", optional: false]}], "hexpm", "ccfb5025179ea307f7f899d1bb3905cd0ac9f687ed77feebc8f67bdca78565c4"},
"telemetry": {:hex, :telemetry, "1.2.1", "68fdfe8d8f05a8428483a97d7aab2f268aaff24b49e0f599faa091f1d4e7f61c", [:rebar3], [], "hexpm", "dad9ce9d8effc621708f99eac538ef1cbe05d6a874dd741de2e689c47feafed5"},
"telemetry": {:hex, :telemetry, "1.3.0", "fedebbae410d715cf8e7062c96a1ef32ec22e764197f70cda73d82778d61e7a2", [:rebar3], [], "hexpm", "7015fc8919dbe63764f4b4b87a95b7c0996bd539e0d499be6ec9d7f3875b79e6"},
"telemetry_metrics": {:hex, :telemetry_metrics, "0.6.2", "2caabe9344ec17eafe5403304771c3539f3b6e2f7fb6a6f602558c825d0d0bfb", [:mix], [{:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "9b43db0dc33863930b9ef9d27137e78974756f5f198cae18409970ed6fa5b561"},
"telemetry_poller": {:hex, :telemetry_poller, "1.0.0", "db91bb424e07f2bb6e73926fcafbfcbcb295f0193e0a00e825e589a0a47e8453", [:rebar3], [{:telemetry, "~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "b3a24eafd66c3f42da30fc3ca7dda1e9d546c12250a2d60d7b81d264fbec4f6e"},
"timex": {:hex, :timex, "3.7.11", "bb95cb4eb1d06e27346325de506bcc6c30f9c6dea40d1ebe390b262fad1862d1", [:mix], [{:combine, "~> 0.10", [hex: :combine, repo: "hexpm", optional: false]}, {:gettext, "~> 0.20", [hex: :gettext, repo: "hexpm", optional: false]}, {:tzdata, "~> 1.1", [hex: :tzdata, repo: "hexpm", optional: false]}], "hexpm", "8b9024f7efbabaf9bd7aa04f65cf8dcd7c9818ca5737677c7b76acbc6a94d1aa"},
"tzdata": {:hex, :tzdata, "1.1.1", "20c8043476dfda8504952d00adac41c6eda23912278add38edc140ae0c5bcc46", [:mix], [{:hackney, "~> 1.17", [hex: :hackney, repo: "hexpm", optional: false]}], "hexpm", "a69cec8352eafcd2e198dea28a34113b60fdc6cb57eb5ad65c10292a6ba89787"},
"tzdata": {:hex, :tzdata, "1.1.2", "45e5f1fcf8729525ec27c65e163be5b3d247ab1702581a94674e008413eef50b", [:mix], [{:hackney, "~> 1.17", [hex: :hackney, repo: "hexpm", optional: false]}], "hexpm", "cec7b286e608371602318c414f344941d5eb0375e14cfdab605cca2fe66cba8b"},
"unicode_util_compat": {:hex, :unicode_util_compat, "0.7.0", "bc84380c9ab48177092f43ac89e4dfa2c6d62b40b8bd132b1059ecc7232f9a78", [:rebar3], [], "hexpm", "25eee6d67df61960cf6a794239566599b09e17e668d3700247bc498638152521"},
"websock": {:hex, :websock, "0.5.3", "2f69a6ebe810328555b6fe5c831a851f485e303a7c8ce6c5f675abeb20ebdadc", [:mix], [], "hexpm", "6105453d7fac22c712ad66fab1d45abdf049868f253cf719b625151460b8b453"},
"websock_adapter": {:hex, :websock_adapter, "0.5.5", "9dfeee8269b27e958a65b3e235b7e447769f66b5b5925385f5a569269164a210", [:mix], [{:bandit, ">= 0.6.0", [hex: :bandit, repo: "hexpm", optional: true]}, {:plug, "~> 1.14", [hex: :plug, repo: "hexpm", optional: false]}, {:plug_cowboy, "~> 2.6", [hex: :plug_cowboy, repo: "hexpm", optional: true]}, {:websock, "~> 0.5", [hex: :websock, repo: "hexpm", optional: false]}], "hexpm", "4b977ba4a01918acbf77045ff88de7f6972c2a009213c515a445c48f224ffce9"},
"websock_adapter": {:hex, :websock_adapter, "0.5.7", "65fa74042530064ef0570b75b43f5c49bb8b235d6515671b3d250022cb8a1f9e", [:mix], [{:bandit, ">= 0.6.0", [hex: :bandit, repo: "hexpm", optional: true]}, {:plug, "~> 1.14", [hex: :plug, repo: "hexpm", optional: false]}, {:plug_cowboy, "~> 2.6", [hex: :plug_cowboy, repo: "hexpm", optional: true]}, {:websock, "~> 0.5", [hex: :websock, repo: "hexpm", optional: false]}], "hexpm", "d0f478ee64deddfec64b800673fd6e0c8888b079d9f3444dd96d2a98383bdbd1"},
}
+2 -2
View File
@@ -6,8 +6,8 @@
},
"scripts": {
"create-erd": "sqleton -o priv/repo/erd.png priv/repo/pinchflat_dev.db",
"lint:check": "prettier . --check --config=tooling/.prettierrc.js --ignore-path=tooling/.prettierignore --ignore-path=.gitignore",
"lint:fix": "prettier . --write --config=tooling/.prettierrc.js --ignore-path=tooling/.prettierignore --ignore-path=.gitignore"
"lint:check": "prettier . --check --config=.prettierrc.js --ignore-path=.prettierignore --ignore-path=.gitignore",
"lint:fix": "prettier . --write --config=.prettierrc.js --ignore-path=.prettierignore --ignore-path=.gitignore"
},
"private": true
}
BIN
View File
Binary file not shown.

Before

Width:  |  Height:  |  Size: 424 KiB

After

Width:  |  Height:  |  Size: 438 KiB

@@ -0,0 +1,13 @@
defmodule Pinchflat.Repo.Migrations.AddMarkedForDeletionAtToSources do
use Ecto.Migration
def change do
alter table(:sources) do
add :marked_for_deletion_at, :utc_datetime
end
alter table(:media_profiles) do
add :marked_for_deletion_at, :utc_datetime
end
end
end
@@ -0,0 +1,9 @@
defmodule Pinchflat.Repo.Migrations.AddUseCookiesToSources do
use Ecto.Migration
def change do
alter table(:sources) do
add :use_cookies, :boolean, default: true, null: false
end
end
end
@@ -0,0 +1,10 @@
defmodule Pinchflat.Repo.Migrations.AddDurationLimitsToSources do
use Ecto.Migration
def change do
alter table(:sources) do
add :min_duration_seconds, :integer
add :max_duration_seconds, :integer
end
end
end
@@ -0,0 +1,9 @@
defmodule Pinchflat.Repo.Migrations.AddMediaContainerToMediaProfiles do
use Ecto.Migration
def change do
alter table(:media_profiles) do
add :media_container, :string
end
end
end
@@ -0,0 +1,9 @@
defmodule Pinchflat.Repo.Migrations.AddPredictedMediaFilepathToMediaItems do
use Ecto.Migration
def change do
alter table(:media_items) do
add :predicted_media_filepath, :string
end
end
end
@@ -0,0 +1,9 @@
defmodule Pinchflat.Repo.Migrations.AddEnabledToSources do
use Ecto.Migration
def change do
alter table(:sources) do
add :enabled, :boolean, default: true, null: false
end
end
end
@@ -0,0 +1,9 @@
defmodule Pinchflat.Repo.Migrations.AddAudioLangToMediaProfiles do
use Ecto.Migration
def change do
alter table(:media_profiles) do
add :audio_track, :string
end
end
end
@@ -9,6 +9,7 @@ defmodule Pinchflat.Boot.PreJobStartupTasksTest do
setup do
stub(YtDlpRunnerMock, :version, fn -> {:ok, "1"} end)
stub(AppriseRunnerMock, :version, fn -> {:ok, "2"} end)
stub(UserScriptRunnerMock, :run, fn _event_type, _data -> {:ok, "3", 0} end)
:ok
end
@@ -112,4 +113,16 @@ defmodule Pinchflat.Boot.PreJobStartupTasksTest do
assert Settings.get!(:apprise_version)
end
end
describe "run_app_init_script" do
test "calls the app_init user script runner" do
expect(UserScriptRunnerMock, :run, fn :app_init, data ->
assert data == %{}
{:ok, "", 0}
end)
PreJobStartupTasks.init(%{})
end
end
end
@@ -6,7 +6,6 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilderTest do
alias Pinchflat.Sources
alias Pinchflat.Profiles
alias Pinchflat.Settings
alias Pinchflat.Utils.FilesystemUtils
alias Pinchflat.Downloading.DownloadOptionBuilder
@@ -252,22 +251,15 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilderTest do
end
end
describe "build/1 when testing quality options" do
test "includes quality options" do
resolutions = ["360", "480", "720", "1080", "2160", "4320"]
describe "build/1 when testing media quality and format options" do
# There are more tests inside QualityOptionBuilderTest
# This is essenitally just testing that we implement that module correctly
Enum.each(resolutions, fn resolution ->
resolution_atom = String.to_existing_atom(resolution <> "p")
test "includes video options for video profiles", %{media_item: media_item} do
assert {:ok, res} = DownloadOptionBuilder.build(media_item)
media_profile = media_profile_fixture(%{preferred_resolution: resolution_atom})
source = source_fixture(%{media_profile_id: media_profile.id})
media_item = Repo.preload(media_item_fixture(source_id: source.id), source: :media_profile)
assert {:ok, res} = DownloadOptionBuilder.build(media_item)
assert {:format_sort, "res:#{resolution},+codec:avc:m4a"} in res
assert {:remux_video, "mp4"} in res
end)
assert {:format_sort, "res:1080,+codec:avc:m4a"} in res
assert {:remux_video, "mp4"} in res
end
test "includes quality options for audio only", %{media_item: media_item} do
@@ -280,17 +272,6 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilderTest do
refute {:remux_video, "mp4"} in res
end
test "includes custom quality options if specified", %{media_item: media_item} do
Settings.set(video_codec_preference: "av01")
Settings.set(audio_codec_preference: "aac")
media_item = update_media_profile_attribute(media_item, %{preferred_resolution: :"1080p"})
assert {:ok, res} = DownloadOptionBuilder.build(media_item)
assert {:format_sort, "res:1080,+codec:av01:aac"} in res
end
end
describe "build/1 when testing sponsorblock options" do
@@ -445,6 +426,22 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilderTest do
end
end
describe "build_quality_options_for/1" do
test "builds quality options for a media item", %{media_item: media_item} do
options = DownloadOptionBuilder.build_quality_options_for(media_item)
assert {:format_sort, "res:1080,+codec:avc:m4a"} in options
assert {:remux_video, "mp4"} in options
end
test "builds quality options for a source", %{media_item: media_item} do
options = DownloadOptionBuilder.build_quality_options_for(media_item.source)
assert {:format_sort, "res:1080,+codec:avc:m4a"} in options
assert {:remux_video, "mp4"} in options
end
end
defp update_media_profile_attribute(media_item_with_preloads, attrs) do
media_item_with_preloads.source.media_profile
|> Profiles.change_media_profile(attrs)
@@ -130,9 +130,6 @@ defmodule Pinchflat.Downloading.DownloadingHelpersTest do
_download_prevented =
media_item_fixture(source_id: source.id, media_filepath: "some/filepath.mp4", prevent_download: true)
_culled =
media_item_fixture(source_id: source.id, media_filepath: "some/filepath.mp4", culled_at: now())
assert [] = DownloadingHelpers.kickoff_redownload_for_existing_media(source)
refute_enqueued(worker: MediaDownloadWorker)
@@ -9,8 +9,9 @@ defmodule Pinchflat.Downloading.MediaDownloadWorkerTest do
alias Pinchflat.Downloading.MediaDownloadWorker
setup do
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot -> {:ok, ""} end)
stub(UserScriptRunnerMock, :run, fn _event_type, _data -> :ok end)
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot -> {:ok, "{}"} end)
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl -> {:ok, ""} end)
stub(UserScriptRunnerMock, :run, fn _event_type, _data -> {:ok, "", 0} end)
stub(HTTPClientMock, :get, fn _url, _headers, _opts -> {:ok, ""} end)
media_item =
@@ -53,10 +54,12 @@ defmodule Pinchflat.Downloading.MediaDownloadWorkerTest do
describe "perform/1" do
test "it saves attributes to the media_item", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl ->
expect(YtDlpRunnerMock, :run, 1, fn _url, _opts, _ot, _addl ->
{:ok, render_metadata(:media_metadata)}
end)
expect(YtDlpRunnerMock, :run, 1, fn _url, _opts, _ot, _addl -> {:ok, ""} end)
assert media_item.media_filepath == nil
perform_job(MediaDownloadWorker, %{id: media_item.id})
media_item = Repo.reload(media_item)
@@ -65,10 +68,12 @@ defmodule Pinchflat.Downloading.MediaDownloadWorkerTest do
end
test "it saves the metadata to the media_item", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl ->
expect(YtDlpRunnerMock, :run, 1, fn _url, _opts, _ot, _addl ->
{:ok, render_metadata(:media_metadata)}
end)
expect(YtDlpRunnerMock, :run, 1, fn _url, _opts, _ot, _addl -> {:ok, ""} end)
assert media_item.metadata == nil
perform_job(MediaDownloadWorker, %{id: media_item.id})
assert Repo.reload(media_item).metadata != nil
@@ -138,13 +143,15 @@ defmodule Pinchflat.Downloading.MediaDownloadWorkerTest do
end
test "it saves the file's size to the database", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl ->
expect(YtDlpRunnerMock, :run, 1, fn _url, _opts, _ot, _addl ->
metadata = render_parsed_metadata(:media_metadata)
FilesystemUtils.write_p!(metadata["filepath"], "test")
{:ok, Phoenix.json_library().encode!(metadata)}
end)
expect(YtDlpRunnerMock, :run, 1, fn _url, _opts, _ot, _addl -> {:ok, ""} end)
perform_job(MediaDownloadWorker, %{id: media_item.id})
media_item = Repo.reload(media_item)
@@ -152,7 +159,7 @@ defmodule Pinchflat.Downloading.MediaDownloadWorkerTest do
end
test "does not set redownloaded_at by default", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl ->
expect(YtDlpRunnerMock, :run, 2, fn _url, _opts, _ot, _addl ->
{:ok, render_metadata(:media_metadata)}
end)
@@ -162,36 +169,38 @@ defmodule Pinchflat.Downloading.MediaDownloadWorkerTest do
assert media_item.media_redownloaded_at == nil
end
test "calls the user script runner", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl ->
{:ok, render_metadata(:media_metadata)}
end)
expect(UserScriptRunnerMock, :run, fn :media_downloaded, data ->
assert data.id == media_item.id
:ok
end)
perform_job(MediaDownloadWorker, %{id: media_item.id})
end
test "does not blow up if the record doesn't exist" do
assert :ok = perform_job(MediaDownloadWorker, %{id: 0})
end
test "sets the no_force_overwrites runner option", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, fn _url, opts, _ot, _addl ->
expect(YtDlpRunnerMock, :run, 1, fn _url, opts, _ot, _addl ->
assert :no_force_overwrites in opts
refute :force_overwrites in opts
{:ok, render_metadata(:media_metadata)}
end)
expect(YtDlpRunnerMock, :run, 1, fn _url, _opts, _ot, _addl -> {:ok, ""} end)
perform_job(MediaDownloadWorker, %{id: media_item.id})
end
end
describe "perform/1 when testing non-downloadable media" do
test "does not retry the job if the media is currently not downloadable", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot ->
{:ok, Phoenix.json_library().encode!(%{"live_status" => "is_live"})}
end)
Oban.Testing.with_testing_mode(:inline, fn ->
{:ok, job} = Oban.insert(MediaDownloadWorker.new(%{id: media_item.id}))
assert job.state == "completed"
end)
end
end
describe "perform/1 when testing forced downloads" do
test "ignores 'prevent_download' if forced", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl -> :ok end)
@@ -203,23 +212,27 @@ defmodule Pinchflat.Downloading.MediaDownloadWorkerTest do
end
test "sets force_overwrites runner option", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, fn _url, opts, _ot, _addl ->
expect(YtDlpRunnerMock, :run, 1, fn _url, opts, _ot, _addl ->
assert :force_overwrites in opts
refute :no_force_overwrites in opts
{:ok, render_metadata(:media_metadata)}
end)
expect(YtDlpRunnerMock, :run, 1, fn _url, _opts, _ot, _addl -> {:ok, ""} end)
perform_job(MediaDownloadWorker, %{id: media_item.id, force: true})
end
end
describe "perform/1 when testing re-downloads" do
test "sets redownloaded_at on the media_item", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl ->
expect(YtDlpRunnerMock, :run, 1, fn _url, _opts, _ot, _addl ->
{:ok, render_metadata(:media_metadata)}
end)
expect(YtDlpRunnerMock, :run, 1, fn _url, _opts, _ot, _addl -> {:ok, ""} end)
perform_job(MediaDownloadWorker, %{id: media_item.id, quality_upgrade?: true})
media_item = Repo.reload(media_item)
@@ -227,14 +240,91 @@ defmodule Pinchflat.Downloading.MediaDownloadWorkerTest do
end
test "sets force_overwrites runner option", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, fn _url, opts, _ot, _addl ->
expect(YtDlpRunnerMock, :run, 1, fn _url, opts, _ot, _addl ->
assert :force_overwrites in opts
refute :no_force_overwrites in opts
{:ok, render_metadata(:media_metadata)}
end)
expect(YtDlpRunnerMock, :run, 1, fn _url, _opts, _ot, _addl -> {:ok, ""} end)
perform_job(MediaDownloadWorker, %{id: media_item.id, force: true})
end
test "deletes old files if the media item has been updated" do
expect(YtDlpRunnerMock, :run, 1, fn _url, _opts, _ot, _addl ->
tmp_media_item = media_item_with_attachments()
metadata = render_parsed_metadata(:media_metadata)
metadata = Map.put(metadata, "filepath", tmp_media_item.media_filepath)
{:ok, Phoenix.json_library().encode!(metadata)}
end)
expect(YtDlpRunnerMock, :run, 1, fn _url, _opts, _ot, _addl -> {:ok, ""} end)
old_media_item = media_item_with_attachments()
perform_job(MediaDownloadWorker, %{id: old_media_item.id, force: true})
updated_media_item = Repo.reload(old_media_item)
assert updated_media_item.media_filepath != old_media_item.media_filepath
refute File.exists?(old_media_item.media_filepath)
assert File.exists?(updated_media_item.media_filepath)
end
end
describe "perform/1 when testing user script callbacks" do
setup do
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl ->
{:ok, render_metadata(:media_metadata)}
end)
:ok
end
test "calls the media_pre_download user script runner", %{media_item: media_item} do
expect(UserScriptRunnerMock, :run, fn :media_pre_download, data ->
assert data.id == media_item.id
{:ok, "", 0}
end)
expect(UserScriptRunnerMock, :run, fn :media_downloaded, _ -> {:ok, "", 0} end)
perform_job(MediaDownloadWorker, %{id: media_item.id})
end
test "does not download the media if the pre-download script returns an error", %{media_item: media_item} do
expect(UserScriptRunnerMock, :run, fn :media_pre_download, _ -> {:ok, "", 1} end)
assert :ok = perform_job(MediaDownloadWorker, %{id: media_item.id})
media_item = Repo.reload!(media_item)
refute media_item.media_filepath
assert media_item.prevent_download
end
test "downloads media if the pre-download script is not present", %{media_item: media_item} do
expect(UserScriptRunnerMock, :run, fn :media_pre_download, _ -> {:ok, :no_executable} end)
expect(UserScriptRunnerMock, :run, fn :media_downloaded, _ -> {:ok, :no_executable} end)
assert :ok = perform_job(MediaDownloadWorker, %{id: media_item.id})
media_item = Repo.reload!(media_item)
assert media_item.media_filepath
refute media_item.prevent_download
end
test "calls the media_downloaded user script runner", %{media_item: media_item} do
expect(UserScriptRunnerMock, :run, fn :media_pre_download, _ -> {:ok, "", 0} end)
expect(UserScriptRunnerMock, :run, fn :media_downloaded, data ->
assert data.id == media_item.id
{:ok, "", 0}
end)
perform_job(MediaDownloadWorker, %{id: media_item.id})
end
end
end
@@ -5,6 +5,7 @@ defmodule Pinchflat.Downloading.MediaDownloaderTest do
import Pinchflat.SourcesFixtures
import Pinchflat.ProfilesFixtures
alias Pinchflat.Media
alias Pinchflat.Downloading.MediaDownloader
setup do
@@ -15,27 +16,30 @@ defmodule Pinchflat.Downloading.MediaDownloaderTest do
)
stub(HTTPClientMock, :get, fn _url, _headers, _opts -> {:ok, ""} end)
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot -> {:ok, ""} end)
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot -> {:ok, "{}"} end)
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl_args -> {:ok, ""} end)
{:ok, %{media_item: media_item}}
end
describe "download_for_media_item/3" do
test "it calls the backend runner", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, fn url, _opts, ot, addl ->
test "calls the backend runner", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, 1, fn url, _opts, ot, addl ->
assert url == media_item.original_url
assert ot == "after_move:%()j"
assert [{:output_filepath, filepath}] = addl
assert [{:output_filepath, filepath} | _] = addl
assert is_binary(filepath)
{:ok, render_metadata(:media_metadata)}
end)
expect(YtDlpRunnerMock, :run, 1, fn _url, _opts, _ot, _addl_args -> {:ok, ""} end)
assert {:ok, _} = MediaDownloader.download_for_media_item(media_item)
end
test "it saves the metadata filepath to the database", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl ->
test "saves the metadata filepath to the database", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, 2, fn _url, _opts, _ot, _addl ->
{:ok, render_metadata(:media_metadata)}
end)
@@ -46,6 +50,14 @@ defmodule Pinchflat.Downloading.MediaDownloaderTest do
assert updated_media_item.metadata.thumbnail_filepath =~ "media_items/#{media_item.id}/thumbnail.jpg"
end
test "errors for non-downloadable media are passed through", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot ->
{:ok, Phoenix.json_library().encode!(%{"live_status" => "is_live"})}
end)
assert {:error, :unsuitable_for_download} = MediaDownloader.download_for_media_item(media_item)
end
test "non-recoverable errors are passed through", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl ->
{:error, :some_error, 1}
@@ -64,21 +76,83 @@ defmodule Pinchflat.Downloading.MediaDownloaderTest do
end
end
describe "download_for_media_item/3 when testing non-downloadable media" do
test "calls the download runner if the media is currently downloadable", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot ->
{:ok, Phoenix.json_library().encode!(%{"live_status" => "was_live"})}
end)
expect(YtDlpRunnerMock, :run, 2, fn _url, _opts, _ot, _addl ->
{:ok, render_metadata(:media_metadata)}
end)
assert {:ok, _} = MediaDownloader.download_for_media_item(media_item)
end
test "does not call the download runner if the media is not downloadable", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot ->
{:ok, Phoenix.json_library().encode!(%{"live_status" => "is_live"})}
end)
expect(YtDlpRunnerMock, :run, 0, fn _url, _opts, _ot, _addl -> {:ok, ""} end)
assert {:error, :unsuitable_for_download} = MediaDownloader.download_for_media_item(media_item)
end
test "returns unexpected errors from the download status determination method", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot -> {:error, :what_tha} end)
assert {:error, "Unknown error: {:error, :what_tha}"} = MediaDownloader.download_for_media_item(media_item)
end
end
describe "download_for_media_item/3 when testing override options" do
test "includes override opts if specified", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, fn _url, opts, _ot, _addl ->
expect(YtDlpRunnerMock, :run, 1, fn _url, opts, _ot, _addl ->
refute :force_overwrites in opts
assert :no_force_overwrites in opts
{:ok, render_metadata(:media_metadata)}
end)
expect(YtDlpRunnerMock, :run, 1, fn _url, _opts, _ot, _addl -> {:ok, ""} end)
override_opts = [overwrite_behaviour: :no_force_overwrites]
assert {:ok, _} = MediaDownloader.download_for_media_item(media_item, override_opts)
end
end
describe "download_for_media_item/3 when testing cookie usage" do
test "sets use_cookies if the source uses cookies" do
expect(YtDlpRunnerMock, :run, 1, fn _url, _opts, _ot, addl ->
assert {:use_cookies, true} in addl
{:ok, render_metadata(:media_metadata)}
end)
expect(YtDlpRunnerMock, :run, 1, fn _url, _opts, _ot, _addl_args -> {:ok, ""} end)
source = source_fixture(%{use_cookies: true})
media_item = media_item_fixture(%{source_id: source.id})
assert {:ok, _} = MediaDownloader.download_for_media_item(media_item)
end
test "does not set use_cookies if the source does not use cookies" do
expect(YtDlpRunnerMock, :run, 1, fn _url, _opts, _ot, addl ->
assert {:use_cookies, false} in addl
{:ok, render_metadata(:media_metadata)}
end)
expect(YtDlpRunnerMock, :run, 1, fn _url, _opts, _ot, _addl_args -> {:ok, ""} end)
source = source_fixture(%{use_cookies: false})
media_item = media_item_fixture(%{source_id: source.id})
assert {:ok, _} = MediaDownloader.download_for_media_item(media_item)
end
end
describe "download_for_media_item/3 when testing retries" do
test "returns a recovered tuple on recoverable errors", %{media_item: media_item} do
message = "Unable to communicate with SponsorBlock"
@@ -93,13 +167,15 @@ defmodule Pinchflat.Downloading.MediaDownloaderTest do
test "attempts to update the media item on recoverable errors", %{media_item: media_item} do
message = "Unable to communicate with SponsorBlock"
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot, addl ->
[{:output_filepath, filepath}] = addl
expect(YtDlpRunnerMock, :run, 1, fn _url, _opts, _ot, addl ->
[{:output_filepath, filepath} | _] = addl
File.write(filepath, render_metadata(:media_metadata))
{:error, message, 1}
end)
expect(YtDlpRunnerMock, :run, 1, fn _url, _opts, _ot, _addl_args -> {:ok, ""} end)
assert {:recovered, ^message} = MediaDownloader.download_for_media_item(media_item)
media_item = Repo.reload(media_item)
@@ -117,42 +193,48 @@ defmodule Pinchflat.Downloading.MediaDownloaderTest do
:ok
end
test "it sets the media_downloaded_at", %{media_item: media_item} do
test "sets the media_downloaded_at", %{media_item: media_item} do
assert media_item.media_downloaded_at == nil
assert {:ok, updated_media_item} = MediaDownloader.download_for_media_item(media_item)
assert DateTime.diff(DateTime.utc_now(), updated_media_item.media_downloaded_at) < 2
end
test "it extracts the title", %{media_item: media_item} do
test "sets the culled_at to nil", %{media_item: media_item} do
Media.update_media_item(media_item, %{culled_at: DateTime.utc_now()})
assert {:ok, updated_media_item} = MediaDownloader.download_for_media_item(media_item)
assert updated_media_item.culled_at == nil
end
test "extracts the title", %{media_item: media_item} do
assert {:ok, updated_media_item} = MediaDownloader.download_for_media_item(media_item)
assert updated_media_item.title == "Pinchflat Example Video"
end
test "it extracts the description", %{media_item: media_item} do
test "extracts the description", %{media_item: media_item} do
assert {:ok, updated_media_item} = MediaDownloader.download_for_media_item(media_item)
assert is_binary(updated_media_item.description)
end
test "it extracts the media_filepath", %{media_item: media_item} do
test "extracts the media_filepath", %{media_item: media_item} do
assert media_item.media_filepath == nil
assert {:ok, updated_media_item} = MediaDownloader.download_for_media_item(media_item)
assert String.ends_with?(updated_media_item.media_filepath, ".mkv")
end
test "it extracts the subtitle_filepaths", %{media_item: media_item} do
test "extracts the subtitle_filepaths", %{media_item: media_item} do
assert media_item.subtitle_filepaths == []
assert {:ok, updated_media_item} = MediaDownloader.download_for_media_item(media_item)
assert [["de", _], ["en", _] | _rest] = updated_media_item.subtitle_filepaths
end
test "it extracts the duration_seconds", %{media_item: media_item} do
test "extracts the duration_seconds", %{media_item: media_item} do
assert media_item.duration_seconds == nil
assert {:ok, updated_media_item} = MediaDownloader.download_for_media_item(media_item)
assert is_integer(updated_media_item.duration_seconds)
end
test "it extracts the thumbnail_filepath", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl ->
test "extracts the thumbnail_filepath", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, 2, fn _url, _opts, _ot, _addl ->
metadata = render_parsed_metadata(:media_metadata)
thumbnail_filepath =
@@ -175,8 +257,8 @@ defmodule Pinchflat.Downloading.MediaDownloaderTest do
File.rm(updated_media_item.thumbnail_filepath)
end
test "it extracts the metadata_filepath", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl ->
test "extracts the metadata_filepath", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, 2, fn _url, _opts, _ot, _addl ->
metadata = render_parsed_metadata(:media_metadata)
infojson_filepath = metadata["infojson_filename"]
@@ -195,14 +277,14 @@ defmodule Pinchflat.Downloading.MediaDownloaderTest do
describe "download_for_media_item/3 when testing NFO generation" do
setup do
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl ->
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl ->
{:ok, render_metadata(:media_metadata)}
end)
:ok
end
test "it generates an NFO file if the source is set to download NFOs" do
test "generates an NFO file if the source is set to download NFOs" do
profile = media_profile_fixture(%{download_nfo: true})
source = source_fixture(%{media_profile_id: profile.id})
media_item = media_item_fixture(%{source_id: source.id})
@@ -215,7 +297,7 @@ defmodule Pinchflat.Downloading.MediaDownloaderTest do
File.rm!(updated_media_item.nfo_filepath)
end
test "it does not generate an NFO file if the source is set to not download NFOs" do
test "does not generate an NFO file if the source is set to not download NFOs" do
profile = media_profile_fixture(%{download_nfo: false})
source = source_fixture(%{media_profile_id: profile.id})
media_item = media_item_fixture(%{source_id: source.id})
@@ -7,15 +7,36 @@ defmodule Pinchflat.Downloading.MediaRetentionWorkerTest do
alias Pinchflat.Media
alias Pinchflat.Downloading.MediaRetentionWorker
describe "perform/1" do
setup do
stub(UserScriptRunnerMock, :run, fn _event_type, _data -> :ok end)
setup do
stub(UserScriptRunnerMock, :run, fn _event_type, _data -> {:ok, "", 0} end)
:ok
:ok
end
describe "perform/1" do
test "sets deleted media to not re-download" do
{_source, old_media_item, new_media_item} = prepare_records_for_retention_date()
perform_job(MediaRetentionWorker, %{})
refute Repo.reload!(new_media_item).prevent_download
assert Repo.reload!(old_media_item).prevent_download
end
test "sets culled_at timestamp on deleted media" do
{_source, old_media_item, new_media_item} = prepare_records_for_retention_date()
perform_job(MediaRetentionWorker, %{})
refute Repo.reload!(new_media_item).culled_at
assert Repo.reload!(old_media_item).culled_at
assert DateTime.diff(now(), Repo.reload!(old_media_item).culled_at) < 1
end
end
describe "perform/1 when testing retention_period-based culling" do
test "deletes media files that are past their retention date" do
{_source, old_media_item, new_media_item} = prepare_records()
{_source, old_media_item, new_media_item} = prepare_records_for_retention_date()
perform_job(MediaRetentionWorker, %{})
@@ -25,27 +46,50 @@ defmodule Pinchflat.Downloading.MediaRetentionWorkerTest do
refute Repo.reload!(old_media_item).media_filepath
end
test "sets deleted media to not re-download" do
{_source, old_media_item, new_media_item} = prepare_records()
test "deletes media files that are on their retention date per the 24-h clock" do
{_source, old_media_item, new_media_item} = prepare_records_for_retention_date(2)
just_over_two_days_ago = now_minus(2, :days) |> DateTime.add(-1, :minute)
just_under_two_days_ago = now_minus(2, :days) |> DateTime.add(1, :minute)
Media.update_media_item(old_media_item, %{media_downloaded_at: just_over_two_days_ago})
Media.update_media_item(new_media_item, %{media_downloaded_at: just_under_two_days_ago})
perform_job(MediaRetentionWorker, %{})
refute Repo.reload!(new_media_item).prevent_download
assert Repo.reload!(old_media_item).prevent_download
assert File.exists?(new_media_item.media_filepath)
refute File.exists?(old_media_item.media_filepath)
assert Repo.reload!(new_media_item).media_filepath
refute Repo.reload!(old_media_item).media_filepath
end
test "sets culled_at timestamp on deleted media" do
{_source, old_media_item, new_media_item} = prepare_records()
test "sets culled_at and prevent_download" do
{_source, old_media_item, new_media_item} = prepare_records_for_retention_date()
perform_job(MediaRetentionWorker, %{})
refute Repo.reload!(new_media_item).culled_at
assert Repo.reload!(old_media_item).culled_at
assert DateTime.diff(now(), Repo.reload!(old_media_item).culled_at) < 1
refute Repo.reload!(new_media_item).prevent_download
assert Repo.reload!(old_media_item).prevent_download
end
test "doesn't cull if the source doesn't have a retention period" do
{_source, old_media_item, new_media_item} = prepare_records_for_retention_date(nil)
perform_job(MediaRetentionWorker, %{})
assert File.exists?(new_media_item.media_filepath)
assert File.exists?(old_media_item.media_filepath)
assert Repo.reload!(new_media_item).media_filepath
assert Repo.reload!(old_media_item).media_filepath
refute Repo.reload!(new_media_item).culled_at
refute Repo.reload!(old_media_item).culled_at
end
test "doesn't cull media items that have prevent_culling set" do
{_source, old_media_item, _new_media_item} = prepare_records()
{_source, old_media_item, _new_media_item} = prepare_records_for_retention_date()
Media.update_media_item(old_media_item, %{prevent_culling: true})
@@ -53,11 +97,101 @@ defmodule Pinchflat.Downloading.MediaRetentionWorkerTest do
assert File.exists?(old_media_item.media_filepath)
assert Repo.reload!(old_media_item).media_filepath
refute Repo.reload!(old_media_item).culled_at
end
test "doesn't cull if the media item has no media_filepath" do
{_source, old_media_item, _new_media_item} = prepare_records_for_retention_date()
Media.update_media_item(old_media_item, %{media_filepath: nil})
perform_job(MediaRetentionWorker, %{})
refute Repo.reload!(old_media_item).culled_at
end
end
defp prepare_records do
source = source_fixture(%{retention_period_days: 2})
describe "perform/1 when testing source cutoff-based culling" do
test "culls media from before the cutoff date" do
{_source, old_media_item, new_media_item} = prepare_records_for_source_cutoff_date()
perform_job(MediaRetentionWorker, %{})
assert File.exists?(new_media_item.media_filepath)
refute File.exists?(old_media_item.media_filepath)
assert Repo.reload!(new_media_item).media_filepath
refute Repo.reload!(old_media_item).media_filepath
end
# NOTE: Since this is a date and not a datetime, we can't add logic to have to-the-minute
# comparison like we can with retention periods. We can only compare to the day.
test "doesn't cull media from on or after the cutoff date" do
{_source, old_media_item, new_media_item} = prepare_records_for_source_cutoff_date(2)
Media.update_media_item(old_media_item, %{uploaded_at: now_minus(2, :days)})
Media.update_media_item(new_media_item, %{uploaded_at: now_minus(1, :day)})
perform_job(MediaRetentionWorker, %{})
assert File.exists?(new_media_item.media_filepath)
assert File.exists?(old_media_item.media_filepath)
assert Repo.reload!(new_media_item).media_filepath
assert Repo.reload!(old_media_item).media_filepath
refute Repo.reload!(new_media_item).culled_at
refute Repo.reload!(old_media_item).culled_at
end
test "sets culled_at but not prevent_download" do
{_source, old_media_item, new_media_item} = prepare_records_for_source_cutoff_date()
perform_job(MediaRetentionWorker, %{})
refute Repo.reload!(new_media_item).culled_at
assert Repo.reload!(old_media_item).culled_at
refute Repo.reload!(new_media_item).prevent_download
refute Repo.reload!(old_media_item).prevent_download
end
test "doesn't cull media if the source doesn't have a cutoff date" do
{_source, old_media_item, new_media_item} = prepare_records_for_source_cutoff_date(nil)
perform_job(MediaRetentionWorker, %{})
assert File.exists?(new_media_item.media_filepath)
assert File.exists?(old_media_item.media_filepath)
assert Repo.reload!(new_media_item).media_filepath
assert Repo.reload!(old_media_item).media_filepath
refute Repo.reload!(new_media_item).culled_at
refute Repo.reload!(old_media_item).culled_at
end
test "doesn't cull media items that have prevent_culling set" do
{_source, old_media_item, _new_media_item} = prepare_records_for_source_cutoff_date()
Media.update_media_item(old_media_item, %{prevent_culling: true})
perform_job(MediaRetentionWorker, %{})
assert File.exists?(old_media_item.media_filepath)
assert Repo.reload!(old_media_item).media_filepath
refute Repo.reload!(old_media_item).culled_at
end
test "doesn't cull if the media item has no media_filepath" do
{_source, old_media_item, _new_media_item} = prepare_records_for_source_cutoff_date()
Media.update_media_item(old_media_item, %{media_filepath: nil})
perform_job(MediaRetentionWorker, %{})
refute Repo.reload!(old_media_item).culled_at
end
end
defp prepare_records_for_retention_date(retention_period_days \\ 2) do
source = source_fixture(%{retention_period_days: retention_period_days})
old_media_item =
media_item_with_attachments(%{
@@ -73,4 +207,23 @@ defmodule Pinchflat.Downloading.MediaRetentionWorkerTest do
{source, old_media_item, new_media_item}
end
defp prepare_records_for_source_cutoff_date(download_cutoff_date_days_ago \\ 2) do
cutoff_date = if download_cutoff_date_days_ago, do: now_minus(download_cutoff_date_days_ago, :days), else: nil
source = source_fixture(%{download_cutoff_date: cutoff_date})
old_media_item =
media_item_with_attachments(%{
source_id: source.id,
uploaded_at: now_minus(3, :days)
})
new_media_item =
media_item_with_attachments(%{
source_id: source.id,
uploaded_at: now_minus(1, :day)
})
{source, old_media_item, new_media_item}
end
end
@@ -0,0 +1,109 @@
defmodule Pinchflat.Downloading.QualityOptionBuilderTest do
use Pinchflat.DataCase
import Pinchflat.ProfilesFixtures
alias Pinchflat.Profiles
alias Pinchflat.Settings
alias Pinchflat.Downloading.QualityOptionBuilder
describe "build/1" do
test "includes format options if audio_track is set to original" do
media_profile = media_profile_fixture(%{audio_track: "original"})
assert res = QualityOptionBuilder.build(media_profile)
assert {:format, "bestvideo+bestaudio[format_note*=original]/bestvideo*+bestaudio/best"} in res
end
test "includes format options if audio_track is set to default" do
media_profile = media_profile_fixture(%{audio_track: "default"})
assert res = QualityOptionBuilder.build(media_profile)
assert {:format, "bestvideo+bestaudio[format_note*='(default)']/bestvideo*+bestaudio/best"} in res
end
test "includes format options if audio_track is set to a language code" do
media_profile = media_profile_fixture(%{audio_track: "en"})
assert res = QualityOptionBuilder.build(media_profile)
assert {:format, "bestvideo+bestaudio[language^=en]/bestvideo*+bestaudio/best"} in res
end
end
describe "build/1 when testing audio profiles" do
setup do
{:ok, media_profile: media_profile_fixture(%{preferred_resolution: :audio})}
end
test "includes quality options for audio only", %{media_profile: media_profile} do
assert res = QualityOptionBuilder.build(media_profile)
assert :extract_audio in res
assert {:format_sort, "+acodec:m4a"} in res
refute {:remux_video, "mp4"} in res
end
test "includes custom format target for audio if specified", %{media_profile: media_profile} do
{:ok, media_profile} =
Profiles.update_media_profile(media_profile, %{media_container: "flac", preferred_resolution: :audio})
assert res = QualityOptionBuilder.build(media_profile)
assert {:audio_format, "flac"} in res
end
test "includes custom format options", %{media_profile: media_profile} do
assert res = QualityOptionBuilder.build(media_profile)
assert {:format, "bestaudio/best"} in res
end
end
describe "build/1 when testing non-audio profiles" do
setup do
{:ok, media_profile: media_profile_fixture(%{preferred_resolution: :"480p"})}
end
test "includes quality options" do
resolutions = ["360", "480", "720", "1080", "2160", "4320"]
Enum.each(resolutions, fn resolution ->
resolution_atom = String.to_existing_atom(resolution <> "p")
media_profile = media_profile_fixture(%{preferred_resolution: resolution_atom})
assert res = QualityOptionBuilder.build(media_profile)
assert {:format_sort, "res:#{resolution},+codec:avc:m4a"} in res
assert {:remux_video, "mp4"} in res
end)
end
test "includes custom quality options if specified", %{media_profile: media_profile} do
Settings.set(video_codec_preference: "av01")
Settings.set(audio_codec_preference: "aac")
{:ok, media_profile} = Profiles.update_media_profile(media_profile, %{preferred_resolution: :"1080p"})
assert res = QualityOptionBuilder.build(media_profile)
assert {:format_sort, "res:1080,+codec:av01:aac"} in res
end
test "includes custom remux target for videos if specified", %{media_profile: media_profile} do
{:ok, media_profile} = Profiles.update_media_profile(media_profile, %{media_container: "mkv"})
assert res = QualityOptionBuilder.build(media_profile)
assert {:remux_video, "mkv"} in res
end
test "includes custom format options", %{media_profile: media_profile} do
assert res = QualityOptionBuilder.build(media_profile)
assert {:format, "bestvideo*+bestaudio/best"} in res
end
end
end
@@ -1,6 +1,7 @@
defmodule Pinchflat.FastIndexing.FastIndexingHelpersTest do
use Pinchflat.DataCase
import Pinchflat.TasksFixtures
import Pinchflat.MediaFixtures
import Pinchflat.SourcesFixtures
import Pinchflat.ProfilesFixtures
@@ -8,17 +9,35 @@ defmodule Pinchflat.FastIndexing.FastIndexingHelpersTest do
alias Pinchflat.Tasks
alias Pinchflat.Settings
alias Pinchflat.Media.MediaItem
alias Pinchflat.FastIndexing.FastIndexingWorker
alias Pinchflat.Downloading.MediaDownloadWorker
alias Pinchflat.FastIndexing.FastIndexingHelpers
setup do
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot ->
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl ->
{:ok, media_attributes_return_fixture()}
end)
{:ok, [source: source_fixture()]}
end
describe "kickoff_indexing_task/1" do
test "deletes any existing fast indexing tasks", %{source: source} do
{:ok, job} = Oban.insert(FastIndexingWorker.new(%{"id" => source.id}))
task = task_fixture(source_id: source.id, job_id: job.id)
assert Repo.reload!(task)
assert {:ok, _} = FastIndexingHelpers.kickoff_indexing_task(source)
assert_raise Ecto.NoResultsError, fn -> Repo.reload!(task) end
end
test "kicks off a new fast indexing task", %{source: source} do
assert {:ok, _} = FastIndexingHelpers.kickoff_indexing_task(source)
assert [worker] = all_enqueued(worker: FastIndexingWorker)
assert worker.args["id"] == source.id
end
end
describe "kickoff_download_tasks_from_youtube_rss_feed/1" do
test "enqueues a new worker for each new media_id in the source's RSS feed", %{source: source} do
expect(HTTPClientMock, :get, fn _url -> {:ok, "<yt:videoId>test_1</yt:videoId>"} end)
@@ -61,19 +80,59 @@ defmodule Pinchflat.FastIndexing.FastIndexingHelpersTest do
assert [_] = Tasks.list_tasks_for(media_item, "MediaDownloadWorker")
end
test "passes the source's download options to the yt-dlp runner", %{source: source} do
expect(HTTPClientMock, :get, fn _url -> {:ok, "<yt:videoId>test_1</yt:videoId>"} end)
expect(YtDlpRunnerMock, :run, fn _url, opts, _ot, _addl_opts ->
assert {:output, "/tmp/test/media/%(title)S.%(ext)S"} in opts
assert {:remux_video, "mp4"} in opts
{:ok, media_attributes_return_fixture()}
end)
FastIndexingHelpers.kickoff_download_tasks_from_youtube_rss_feed(source)
end
test "sets use_cookies if the source uses cookies" do
expect(HTTPClientMock, :get, fn _url -> {:ok, "<yt:videoId>test_1</yt:videoId>"} end)
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot, addl ->
assert {:use_cookies, true} in addl
{:ok, media_attributes_return_fixture()}
end)
source = source_fixture(%{use_cookies: true})
assert [%MediaItem{}] = FastIndexingHelpers.kickoff_download_tasks_from_youtube_rss_feed(source)
end
test "does not set use_cookies if the source does not use cookies" do
expect(HTTPClientMock, :get, fn _url -> {:ok, "<yt:videoId>test_1</yt:videoId>"} end)
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot, addl ->
assert {:use_cookies, false} in addl
{:ok, media_attributes_return_fixture()}
end)
source = source_fixture(%{use_cookies: false})
assert [%MediaItem{}] = FastIndexingHelpers.kickoff_download_tasks_from_youtube_rss_feed(source)
end
test "does not enqueue a download job if the media item does not match the format rules" do
expect(HTTPClientMock, :get, fn _url -> {:ok, "<yt:videoId>test_1</yt:videoId>"} end)
profile = media_profile_fixture(%{shorts_behaviour: :exclude})
source = source_fixture(%{media_profile_id: profile.id})
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot ->
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl ->
output =
Phoenix.json_library().encode!(%{
id: "video2",
title: "Video 2",
webpage_url: "https://example.com/shorts/video2",
was_live: true,
original_url: "https://example.com/shorts/video2",
live_status: "is_live",
description: "desc2",
aspect_ratio: 1.67,
duration: 345.67,
@@ -91,7 +150,7 @@ defmodule Pinchflat.FastIndexing.FastIndexingHelpersTest do
test "does not blow up if a media item cannot be created", %{source: source} do
expect(HTTPClientMock, :get, fn _url -> {:ok, "<yt:videoId>test_1</yt:videoId>"} end)
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot ->
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl ->
{:ok, "{}"}
end)
@@ -101,7 +160,7 @@ defmodule Pinchflat.FastIndexing.FastIndexingHelpersTest do
test "does not blow up if a media item causes a yt-dlp error", %{source: source} do
expect(HTTPClientMock, :get, fn _url -> {:ok, "<yt:videoId>test_1</yt:videoId>"} end)
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot ->
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl ->
{:error, "message", 1}
end)
@@ -84,7 +84,7 @@ defmodule Pinchflat.FastIndexing.FastIndexingWorkerTest do
source = source_fixture(fast_index: true)
expect(HTTPClientMock, :get, fn _url -> {:ok, "<yt:videoId>test_1</yt:videoId>"} end)
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot -> {:ok, render_metadata(:media_metadata)} end)
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl -> {:ok, render_metadata(:media_metadata)} end)
expect(AppriseRunnerMock, :run, fn servers, opts ->
assert "server_1" = servers
@@ -110,7 +110,7 @@ defmodule Pinchflat.FastIndexing.FastIndexingWorkerTest do
source = source_fixture(fast_index: true, download_media: false)
expect(HTTPClientMock, :get, fn _url -> {:ok, "<yt:videoId>test_1</yt:videoId>"} end)
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot -> {:ok, render_metadata(:media_metadata)} end)
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl -> {:ok, render_metadata(:media_metadata)} end)
expect(AppriseRunnerMock, :run, 0, fn _servers, _opts -> {:ok, ""} end)
perform_job(FastIndexingWorker, %{id: source.id})
@@ -120,7 +120,7 @@ defmodule Pinchflat.FastIndexing.FastIndexingWorkerTest do
source = source_fixture(fast_index: true, title_filter_regex: "foobar")
expect(HTTPClientMock, :get, fn _url -> {:ok, "<yt:videoId>test_1</yt:videoId>"} end)
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot -> {:ok, render_metadata(:media_metadata)} end)
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl -> {:ok, render_metadata(:media_metadata)} end)
expect(AppriseRunnerMock, :run, 0, fn _servers, _opts -> {:ok, ""} end)
perform_job(FastIndexingWorker, %{id: source.id})
@@ -19,7 +19,7 @@ defmodule Pinchflat.Lifecycle.UserScripts.CommandRunnerTest do
File.write(filepath(), "#!/bin/bash\ntouch #{filename}\n")
refute File.exists?(filename)
assert :ok = Runner.run(:media_downloaded, %{})
assert {:ok, _, _} = Runner.run(:media_downloaded, %{})
assert File.exists?(filename)
end
@@ -27,7 +27,7 @@ defmodule Pinchflat.Lifecycle.UserScripts.CommandRunnerTest do
tmp_dir = Application.get_env(:pinchflat, :tmpfile_directory)
File.write(filepath(), "#!/bin/bash\necho $1 > #{tmp_dir}/event_name\n")
assert :ok = Runner.run(:media_downloaded, %{})
assert {:ok, _, _} = Runner.run(:media_downloaded, %{})
assert File.read!("#{tmp_dir}/event_name") == "media_downloaded\n"
end
@@ -35,26 +35,32 @@ defmodule Pinchflat.Lifecycle.UserScripts.CommandRunnerTest do
tmp_dir = Application.get_env(:pinchflat, :tmpfile_directory)
File.write(filepath(), "#!/bin/bash\necho $2 > #{tmp_dir}/encoded_data\n")
assert :ok = Runner.run(:media_downloaded, %{foo: "bar"})
assert {:ok, _, _} = Runner.run(:media_downloaded, %{foo: "bar"})
assert File.read!("#{tmp_dir}/encoded_data") == "{\"foo\":\"bar\"}\n"
end
test "does nothing if the lifecycle file is not present" do
:ok = File.rm(filepath())
assert :ok = Runner.run(:media_downloaded, %{})
assert {:ok, :no_executable} = Runner.run(:media_downloaded, %{})
end
test "does nothing if the lifecycle file is empty" do
File.write(filepath(), "")
assert :ok = Runner.run(:media_downloaded, %{})
assert {:ok, :no_executable} = Runner.run(:media_downloaded, %{})
end
test "returns :ok if the command exits with a non-zero status" do
File.write(filepath(), "#!/bin/bash\nexit 1\n")
assert :ok = Runner.run(:media_downloaded, %{})
assert {:ok, _, 1} = Runner.run(:media_downloaded, %{})
end
test "returns the output of the command" do
File.write(filepath(), "#!/bin/bash\necho 'hello'\n")
assert {:ok, "hello\n", 0} = Runner.run(:media_downloaded, %{})
end
test "gets upset if you pass an invalid event type" do
+122
View File
@@ -0,0 +1,122 @@
defmodule Pinchflat.Media.FileSyncingTest do
use Pinchflat.DataCase
import Pinchflat.MediaFixtures
alias Pinchflat.Media.FileSyncing
describe "delete_outdated_files/2" do
test "deletes outdated non-subtitle files" do
new_media_item = media_item_with_attachments()
old_media_item = media_item_with_attachments()
assert :ok = FileSyncing.delete_outdated_files(old_media_item, new_media_item)
assert File.exists?(new_media_item.media_filepath)
refute File.exists?(old_media_item.media_filepath)
end
test "doesn't delete non-subtitle files if the new file is the same" do
new_media_item = media_item_with_attachments()
old_media_item = media_item_fixture(%{media_filepath: new_media_item.media_filepath})
assert :ok = FileSyncing.delete_outdated_files(old_media_item, new_media_item)
assert File.exists?(new_media_item.media_filepath)
assert File.exists?(old_media_item.media_filepath)
end
test "doesn't delete the old file if the new file is missing that key" do
new_media_item = media_item_fixture(%{media_filepath: nil})
old_media_item = media_item_with_attachments()
assert :ok = FileSyncing.delete_outdated_files(old_media_item, new_media_item)
assert File.exists?(old_media_item.media_filepath)
end
test "deletes outdated subtitle files" do
new_media_item = media_item_with_attachments()
old_media_item = media_item_with_attachments()
assert :ok = FileSyncing.delete_outdated_files(old_media_item, new_media_item)
assert File.exists?(get_subtitle_filepath(new_media_item, "en"))
refute File.exists?(get_subtitle_filepath(old_media_item, "en"))
end
test "keeps old subtitle files if the new file is the same" do
new_media_item = media_item_with_attachments()
old_media_item = media_item_fixture(%{subtitle_filepaths: new_media_item.subtitle_filepaths})
assert :ok = FileSyncing.delete_outdated_files(old_media_item, new_media_item)
assert File.exists?(get_subtitle_filepath(new_media_item, "en"))
assert File.exists?(get_subtitle_filepath(old_media_item, "en"))
end
test "doesn't delete old subtitle files if the new file is missing that key" do
new_media_item = media_item_fixture(%{subtitle_filepaths: []})
old_media_item = media_item_with_attachments()
assert :ok = FileSyncing.delete_outdated_files(old_media_item, new_media_item)
assert File.exists?(get_subtitle_filepath(old_media_item, "en"))
end
end
describe "sync_file_presence_on_disk/1" do
test "removes attributes whose files are missing" do
media_item = media_item_fixture(%{media_filepath: "/tmp/missing_file.mp4"})
assert media_item.media_filepath
assert [updated_media_item] = FileSyncing.sync_file_presence_on_disk([media_item])
refute updated_media_item.media_filepath
end
test "doesn't remove attributes where the files still exist" do
media_item = media_item_with_attachments()
assert media_item.media_filepath
assert [updated_media_item] = FileSyncing.sync_file_presence_on_disk([media_item])
assert updated_media_item.media_filepath
end
test "doesn't touch other attributes if some are missing and some aren't" do
media_item = media_item_with_attachments()
File.rm(media_item.media_filepath)
assert media_item.thumbnail_filepath
assert media_item.media_filepath
assert [updated_media_item] = FileSyncing.sync_file_presence_on_disk([media_item])
assert updated_media_item.thumbnail_filepath
refute updated_media_item.media_filepath
end
test "removes subtitle files that are missing" do
media_item = media_item_fixture(%{subtitle_filepaths: [["en", "/tmp/missing_file.srt"]]})
assert get_subtitle_filepath(media_item, "en")
assert [updated_media_item] = FileSyncing.sync_file_presence_on_disk([media_item])
refute get_subtitle_filepath(updated_media_item, "en")
end
test "doesn't remove subtitle files that still exist" do
media_item = media_item_with_attachments()
assert get_subtitle_filepath(media_item, "en")
assert [updated_media_item] = FileSyncing.sync_file_presence_on_disk([media_item])
assert get_subtitle_filepath(updated_media_item, "en")
end
end
defp get_subtitle_filepath(media_item, language) do
Enum.reduce_while(media_item.subtitle_filepaths, nil, fn [lang, filepath], acc ->
if lang == language do
{:halt, filepath}
else
{:cont, acc}
end
end)
end
end
@@ -0,0 +1,37 @@
defmodule Pinchflat.Media.FileSyncingWorkerTest do
use Pinchflat.DataCase
import Pinchflat.MediaFixtures
import Pinchflat.SourcesFixtures
alias Pinchflat.Media.FileSyncingWorker
describe "kickoff_with_task/3" do
test "starts the worker" do
source = source_fixture()
assert [] = all_enqueued(worker: FileSyncingWorker)
assert {:ok, _} = FileSyncingWorker.kickoff_with_task(source)
assert [_] = all_enqueued(worker: FileSyncingWorker)
end
test "attaches a task" do
source = source_fixture()
assert {:ok, task} = FileSyncingWorker.kickoff_with_task(source)
assert task.source_id == source.id
end
end
describe "perform/1" do
test "syncs file presence on disk" do
source = source_fixture()
media_item = media_item_fixture(%{media_filepath: "/tmp/missing.mp4", source_id: source.id})
perform_job(FileSyncingWorker, %{"id" => source.id})
updated_media_item = Repo.reload!(media_item)
refute updated_media_item.media_filepath
end
end
end
+113 -113
View File
@@ -41,99 +41,7 @@ defmodule Pinchflat.MediaTest do
end
end
describe "list_cullable_media_items/0" do
test "returns media items where the source has a retention period" do
source_one = source_fixture(%{retention_period_days: 2})
source_two = source_fixture(%{retention_period_days: 0})
source_three = source_fixture(%{retention_period_days: nil})
_media_item =
media_item_fixture(%{
source_id: source_two.id,
media_filepath: "/video/#{Faker.File.file_name(:video)}",
media_downloaded_at: now_minus(3, :days)
})
_media_item =
media_item_fixture(%{
source_id: source_three.id,
media_filepath: "/video/#{Faker.File.file_name(:video)}",
media_downloaded_at: now_minus(3, :days)
})
expected_media_item =
media_item_fixture(%{
source_id: source_one.id,
media_filepath: "/video/#{Faker.File.file_name(:video)}",
media_downloaded_at: now_minus(3, :days)
})
assert Media.list_cullable_media_items() == [expected_media_item]
end
test "returns media_items with a media_filepath" do
source = source_fixture(%{retention_period_days: 2})
_media_item =
media_item_fixture(%{
source_id: source.id,
media_filepath: nil,
media_downloaded_at: now_minus(3, :days)
})
expected_media_item =
media_item_fixture(%{
source_id: source.id,
media_filepath: "/video/#{Faker.File.file_name(:video)}",
media_downloaded_at: now_minus(3, :days)
})
assert Media.list_cullable_media_items() == [expected_media_item]
end
test "returns items that have passed their retention period" do
source = source_fixture(%{retention_period_days: 2})
_media_item =
media_item_fixture(%{
source_id: source.id,
media_filepath: "/video/#{Faker.File.file_name(:video)}",
media_downloaded_at: now_minus(2, :days)
})
expected_media_item =
media_item_fixture(%{
source_id: source.id,
media_filepath: "/video/#{Faker.File.file_name(:video)}",
media_downloaded_at: now_minus(3, :days)
})
assert Media.list_cullable_media_items() == [expected_media_item]
end
test "doesn't return items that are set to prevent culling" do
source = source_fixture(%{retention_period_days: 2})
_media_item =
media_item_fixture(%{
source_id: source.id,
media_filepath: "/video/#{Faker.File.file_name(:video)}",
media_downloaded_at: now_minus(3, :days),
prevent_culling: true
})
expected_media_item =
media_item_fixture(%{
source_id: source.id,
media_filepath: "/video/#{Faker.File.file_name(:video)}",
media_downloaded_at: now_minus(3, :days)
})
assert Media.list_cullable_media_items() == [expected_media_item]
end
end
describe "list_redownloadable_media_items/0" do
describe "list_upgradeable_media_items/0" do
setup do
media_profile = media_profile_fixture(%{redownload_delay_days: 4})
source = source_fixture(%{media_profile_id: media_profile.id, inserted_at: now_minus(10, :days)})
@@ -149,7 +57,7 @@ defmodule Pinchflat.MediaTest do
media_downloaded_at: now_minus(5, :days)
})
assert Media.list_redownloadable_media_items() == [media_item]
assert Media.list_upgradeable_media_items() == [media_item]
end
test "returns media items that were downloaded in past but still meet redownload delay", %{source: source} do
@@ -160,7 +68,7 @@ defmodule Pinchflat.MediaTest do
media_downloaded_at: now_minus(19, :days)
})
assert Media.list_redownloadable_media_items() == [media_item]
assert Media.list_upgradeable_media_items() == [media_item]
end
test "does not return media items without a media_downloaded_at", %{source: source} do
@@ -171,7 +79,7 @@ defmodule Pinchflat.MediaTest do
media_downloaded_at: nil
})
assert Media.list_redownloadable_media_items() == []
assert Media.list_upgradeable_media_items() == []
end
test "does not return media items that are set to prevent download", %{source: source} do
@@ -183,7 +91,7 @@ defmodule Pinchflat.MediaTest do
prevent_download: true
})
assert Media.list_redownloadable_media_items() == []
assert Media.list_upgradeable_media_items() == []
end
test "does not return media items that have been culled", %{source: source} do
@@ -195,7 +103,7 @@ defmodule Pinchflat.MediaTest do
culled_at: now()
})
assert Media.list_redownloadable_media_items() == []
assert Media.list_upgradeable_media_items() == []
end
test "does not return media items before the download delay", %{source: source} do
@@ -206,7 +114,7 @@ defmodule Pinchflat.MediaTest do
media_downloaded_at: now_minus(3, :days)
})
assert Media.list_redownloadable_media_items() == []
assert Media.list_upgradeable_media_items() == []
end
test "does not return media items that have already been redownloaded", %{source: source} do
@@ -218,7 +126,7 @@ defmodule Pinchflat.MediaTest do
media_redownloaded_at: now()
})
assert Media.list_redownloadable_media_items() == []
assert Media.list_upgradeable_media_items() == []
end
test "does not return media items that were first downloaded well after the uploaded_at", %{source: source} do
@@ -229,7 +137,7 @@ defmodule Pinchflat.MediaTest do
uploaded_at: now_minus(20, :days)
})
assert Media.list_redownloadable_media_items() == []
assert Media.list_upgradeable_media_items() == []
end
test "does not return media items that were recently uploaded", %{source: source} do
@@ -240,7 +148,7 @@ defmodule Pinchflat.MediaTest do
uploaded_at: now_minus(2, :days)
})
assert Media.list_redownloadable_media_items() == []
assert Media.list_upgradeable_media_items() == []
end
test "does not return media items without a redownload delay" do
@@ -254,7 +162,7 @@ defmodule Pinchflat.MediaTest do
media_downloaded_at: now_minus(5, :days)
})
assert Media.list_redownloadable_media_items() == []
assert Media.list_upgradeable_media_items() == []
end
end
@@ -447,6 +355,48 @@ defmodule Pinchflat.MediaTest do
end
end
describe "list_pending_media_items_for/1 when min and max durations" do
test "returns media items that meet the min and max duration" do
source = source_fixture(%{min_duration_seconds: 10, max_duration_seconds: 20})
_short_media_item = media_item_fixture(%{source_id: source.id, media_filepath: nil, duration_seconds: 5})
normal_media_item = media_item_fixture(%{source_id: source.id, media_filepath: nil, duration_seconds: 15})
_long_media_item = media_item_fixture(%{source_id: source.id, media_filepath: nil, duration_seconds: 25})
assert Media.list_pending_media_items_for(source) == [normal_media_item]
end
test "does not apply a min duration if none is specified" do
source = source_fixture(%{min_duration_seconds: nil, max_duration_seconds: 20})
short_media_item = media_item_fixture(%{source_id: source.id, media_filepath: nil, duration_seconds: 5})
normal_media_item = media_item_fixture(%{source_id: source.id, media_filepath: nil, duration_seconds: 15})
_long_media_item = media_item_fixture(%{source_id: source.id, media_filepath: nil, duration_seconds: 25})
assert Media.list_pending_media_items_for(source) == [short_media_item, normal_media_item]
end
test "does not apply a max duration if none is specified" do
source = source_fixture(%{min_duration_seconds: 10, max_duration_seconds: nil})
_short_media_item = media_item_fixture(%{source_id: source.id, media_filepath: nil, duration_seconds: 5})
normal_media_item = media_item_fixture(%{source_id: source.id, media_filepath: nil, duration_seconds: 15})
long_media_item = media_item_fixture(%{source_id: source.id, media_filepath: nil, duration_seconds: 25})
assert Media.list_pending_media_items_for(source) == [normal_media_item, long_media_item]
end
test "does not apply a min or max duration if none are specified" do
source = source_fixture(%{min_duration_seconds: nil, max_duration_seconds: nil})
short_media_item = media_item_fixture(%{source_id: source.id, media_filepath: nil, duration_seconds: 5})
normal_media_item = media_item_fixture(%{source_id: source.id, media_filepath: nil, duration_seconds: 15})
long_media_item = media_item_fixture(%{source_id: source.id, media_filepath: nil, duration_seconds: 25})
assert Media.list_pending_media_items_for(source) == [short_media_item, normal_media_item, long_media_item]
end
end
describe "list_pending_media_items_for/1 when testing download prevention" do
test "returns only media items that are not prevented from downloading" do
source = source_fixture()
@@ -491,6 +441,13 @@ defmodule Pinchflat.MediaTest do
assert Media.pending_download?(media_item)
end
test "returns true if the cutoff date is equal to the upload date" do
source = source_fixture(%{download_cutoff_date: now_minus(2, :days)})
media_item = media_item_fixture(%{source_id: source.id, media_filepath: nil, uploaded_at: now_minus(2, :days)})
assert Media.pending_download?(media_item)
end
test "returns false if there is a cutoff date after the media's upload date" do
source = source_fixture(%{download_cutoff_date: now_minus(1, :day)})
media_item = media_item_fixture(%{source_id: source.id, media_filepath: nil, uploaded_at: now_minus(2, :days)})
@@ -526,6 +483,34 @@ defmodule Pinchflat.MediaTest do
assert Media.pending_download?(media_item)
end
test "returns true if the duration is between the min and max" do
source = source_fixture(%{min_duration_seconds: 10, max_duration_seconds: 20})
media_item = media_item_fixture(%{source_id: source.id, media_filepath: nil, duration_seconds: 15})
assert Media.pending_download?(media_item)
end
test "returns false if the duration is below the min" do
source = source_fixture(%{min_duration_seconds: 10, max_duration_seconds: 20})
media_item = media_item_fixture(%{source_id: source.id, media_filepath: nil, duration_seconds: 5})
refute Media.pending_download?(media_item)
end
test "returns false if the duration is above the max" do
source = source_fixture(%{min_duration_seconds: 10, max_duration_seconds: 20})
media_item = media_item_fixture(%{source_id: source.id, media_filepath: nil, duration_seconds: 25})
refute Media.pending_download?(media_item)
end
test "returns true if there is no min or max duration" do
source = source_fixture(%{min_duration_seconds: nil, max_duration_seconds: nil})
media_item = media_item_fixture(%{source_id: source.id, media_filepath: nil, duration_seconds: 15})
assert Media.pending_download?(media_item)
end
test "returns true if the media item is not prevented from downloading" do
media_item = media_item_fixture(%{media_filepath: nil, prevent_download: false})
@@ -727,6 +712,21 @@ defmodule Pinchflat.MediaTest do
assert media_item.media_filepath == update_attrs.media_filepath
end
test "updating strips playlist_index from the provided attrs" do
media_item = media_item_fixture(playlist_index: 5)
update_attrs = %{
media_id: Faker.String.base64(12),
title: Faker.Commerce.product_name(),
media_filepath: "/video/#{Faker.File.file_name(:video)}",
source_id: source_fixture().id,
playlist_index: 1
}
assert {:ok, %MediaItem{} = media_item} = Media.update_media_item(media_item, update_attrs)
assert media_item.playlist_index == 5
end
test "updating with invalid data returns error changeset" do
media_item = media_item_fixture()
assert {:error, %Ecto.Changeset{}} = Media.update_media_item(media_item, @invalid_attrs)
@@ -757,8 +757,8 @@ defmodule Pinchflat.MediaTest do
end
test "does delete the media item's metadata files" do
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot -> {:ok, ""} end)
media_item = Repo.preload(media_item_with_attachments(), :metadata)
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl -> {:ok, ""} end)
media_item = Repo.preload(media_item_with_attachments(), [:metadata, :source])
update_attrs = %{
metadata: %{
@@ -776,7 +776,7 @@ defmodule Pinchflat.MediaTest do
describe "delete_media_item/2 when testing file deletion" do
setup do
stub(UserScriptRunnerMock, :run, fn _event_type, _data -> :ok end)
stub(UserScriptRunnerMock, :run, fn _event_type, _data -> {:ok, "", 0} end)
:ok
end
@@ -789,8 +789,8 @@ defmodule Pinchflat.MediaTest do
end
test "deletes the media item's metadata files" do
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot -> {:ok, ""} end)
media_item = Repo.preload(media_item_with_attachments(), :metadata)
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl -> {:ok, ""} end)
media_item = Repo.preload(media_item_with_attachments(), [:metadata, :source])
update_attrs = %{
metadata: %{
@@ -837,7 +837,7 @@ defmodule Pinchflat.MediaTest do
expect(UserScriptRunnerMock, :run, fn :media_deleted, data ->
assert data.id == media_item.id
:ok
{:ok, "", 0}
end)
assert {:ok, _} = Media.delete_media_item(media_item, delete_files: true)
@@ -846,7 +846,7 @@ defmodule Pinchflat.MediaTest do
describe "delete_media_files/2" do
setup do
stub(UserScriptRunnerMock, :run, fn _event_type, _data -> :ok end)
stub(UserScriptRunnerMock, :run, fn _event_type, _data -> {:ok, "", 0} end)
:ok
end
@@ -875,8 +875,8 @@ defmodule Pinchflat.MediaTest do
end
test "does not delete the media item's metadata files" do
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot -> {:ok, ""} end)
media_item = Repo.preload(media_item_with_attachments(), :metadata)
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl -> {:ok, ""} end)
media_item = Repo.preload(media_item_with_attachments(), [:metadata, :source])
update_attrs = %{
metadata: %{
@@ -909,7 +909,7 @@ defmodule Pinchflat.MediaTest do
expect(UserScriptRunnerMock, :run, fn :media_deleted, data ->
assert data.id == media_item.id
:ok
{:ok, "", 0}
end)
assert {:ok, _} = Media.delete_media_files(media_item)
@@ -2,11 +2,12 @@ defmodule Pinchflat.Metadata.MetadataFileHelpersTest do
use Pinchflat.DataCase
import Pinchflat.MediaFixtures
import Pinchflat.SourcesFixtures
alias Pinchflat.Metadata.MetadataFileHelpers, as: Helpers
setup do
media_item = media_item_fixture()
media_item = Repo.preload(media_item_fixture(), :source)
{:ok, %{media_item: media_item}}
end
@@ -61,7 +62,7 @@ defmodule Pinchflat.Metadata.MetadataFileHelpersTest do
describe "download_and_store_thumbnail_for/2" do
test "returns the filepath", %{media_item: media_item} do
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot -> {:ok, ""} end)
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl -> {:ok, ""} end)
filepath = Helpers.download_and_store_thumbnail_for(media_item)
@@ -69,7 +70,7 @@ defmodule Pinchflat.Metadata.MetadataFileHelpersTest do
end
test "calls yt-dlp with the expected options", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, fn url, opts, ot ->
expect(YtDlpRunnerMock, :run, fn url, opts, ot, _addl ->
assert url == media_item.original_url
assert ot == "after_move:%()j"
@@ -87,8 +88,32 @@ defmodule Pinchflat.Metadata.MetadataFileHelpersTest do
Helpers.download_and_store_thumbnail_for(media_item)
end
test "sets use_cookies if the source uses cookies" do
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot, addl ->
assert {:use_cookies, true} in addl
{:ok, ""}
end)
source = source_fixture(%{use_cookies: true})
media_item = Repo.preload(media_item_fixture(%{source_id: source.id}), :source)
Helpers.download_and_store_thumbnail_for(media_item)
end
test "does not set use_cookies if the source does not use cookies" do
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot, addl ->
assert {:use_cookies, false} in addl
{:ok, ""}
end)
source = source_fixture(%{use_cookies: false})
media_item = Repo.preload(media_item_fixture(%{source_id: source.id}), :source)
Helpers.download_and_store_thumbnail_for(media_item)
end
test "returns nil if yt-dlp fails", %{media_item: media_item} do
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot -> {:error, "error"} end)
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl -> {:error, "error"} end)
filepath = Helpers.download_and_store_thumbnail_for(media_item)
@@ -43,11 +43,12 @@ defmodule Pinchflat.Metadata.MetadataParserTest do
test "it extracts the livestream flag", %{metadata: metadata} do
result = Parser.parse_for_media_item(metadata)
assert result.livestream == metadata["was_live"]
assert metadata["live_status"] == "not_live"
refute result.livestream
end
test "the livestream flag defaults to false", %{metadata: metadata} do
metadata = Map.put(metadata, "was_live", nil)
metadata = Map.put(metadata, "live_status", nil)
result = Parser.parse_for_media_item(metadata)
@@ -32,8 +32,8 @@ defmodule Pinchflat.Metadata.SourceMetadataStorageWorkerTest do
describe "perform/1" do
test "won't call itself in an infinite loop" do
stub(YtDlpRunnerMock, :run, fn
_url, _opts, ot when ot == @source_details_ot -> {:ok, source_details_return_fixture()}
_url, _opts, ot when ot == @metadata_ot -> {:ok, "{}"}
_url, _opts, ot, _addl when ot == @source_details_ot -> {:ok, source_details_return_fixture()}
_url, _opts, ot, _addl when ot == @metadata_ot -> {:ok, "{}"}
end)
source = source_fixture()
@@ -51,8 +51,8 @@ defmodule Pinchflat.Metadata.SourceMetadataStorageWorkerTest do
describe "perform/1 when testing attribute updates" do
test "the source description is saved" do
stub(YtDlpRunnerMock, :run, fn
_url, _opts, ot when ot == @source_details_ot -> {:ok, source_details_return_fixture()}
_url, _opts, ot when ot == @metadata_ot -> {:ok, render_metadata(:channel_source_metadata)}
_url, _opts, ot, _addl when ot == @source_details_ot -> {:ok, source_details_return_fixture()}
_url, _opts, ot, _addl when ot == @metadata_ot -> {:ok, render_metadata(:channel_source_metadata)}
end)
source = source_fixture(%{description: nil})
@@ -68,8 +68,8 @@ defmodule Pinchflat.Metadata.SourceMetadataStorageWorkerTest do
describe "perform/1 when testing metadata storage" do
test "sets metadata location for source" do
stub(YtDlpRunnerMock, :run, fn
_url, _opts, ot when ot == @source_details_ot -> {:ok, source_details_return_fixture()}
_url, _opts, ot when ot == @metadata_ot -> {:ok, "{}"}
_url, _opts, ot, _addl when ot == @source_details_ot -> {:ok, source_details_return_fixture()}
_url, _opts, ot, _addl when ot == @metadata_ot -> {:ok, "{}"}
end)
source = Repo.preload(source_fixture(), :metadata)
@@ -87,8 +87,8 @@ defmodule Pinchflat.Metadata.SourceMetadataStorageWorkerTest do
file_contents = Phoenix.json_library().encode!(%{"title" => "test"})
stub(YtDlpRunnerMock, :run, fn
_url, _opts, ot when ot == @source_details_ot -> {:ok, source_details_return_fixture()}
_url, _opts, ot when ot == @metadata_ot -> {:ok, file_contents}
_url, _opts, ot, _addl when ot == @source_details_ot -> {:ok, source_details_return_fixture()}
_url, _opts, ot, _addl when ot == @metadata_ot -> {:ok, file_contents}
end)
perform_job(SourceMetadataStorageWorker, %{id: source.id})
@@ -100,8 +100,8 @@ defmodule Pinchflat.Metadata.SourceMetadataStorageWorkerTest do
test "sets metadata image location for source" do
stub(YtDlpRunnerMock, :run, fn
_url, _opts, ot when ot == @source_details_ot -> {:ok, source_details_return_fixture()}
_url, _opts, ot when ot == @metadata_ot -> {:ok, render_metadata(:channel_source_metadata)}
_url, _opts, ot, _addl when ot == @source_details_ot -> {:ok, source_details_return_fixture()}
_url, _opts, ot, _addl when ot == @metadata_ot -> {:ok, render_metadata(:channel_source_metadata)}
end)
source = source_fixture()
@@ -118,8 +118,8 @@ defmodule Pinchflat.Metadata.SourceMetadataStorageWorkerTest do
test "stores metadata images for source" do
stub(YtDlpRunnerMock, :run, fn
_url, _opts, ot when ot == @source_details_ot -> {:ok, source_details_return_fixture()}
_url, _opts, ot when ot == @metadata_ot -> {:ok, render_metadata(:channel_source_metadata)}
_url, _opts, ot, _addl when ot == @source_details_ot -> {:ok, source_details_return_fixture()}
_url, _opts, ot, _addl when ot == @metadata_ot -> {:ok, render_metadata(:channel_source_metadata)}
end)
source = source_fixture()
@@ -138,12 +138,12 @@ defmodule Pinchflat.Metadata.SourceMetadataStorageWorkerTest do
describe "perform/1 when testing source image downloading" do
test "downloads and stores source images" do
stub(YtDlpRunnerMock, :run, fn
_url, _opts, ot when ot == @source_details_ot ->
_url, _opts, ot, _addl when ot == @source_details_ot ->
filename = Path.join([Application.get_env(:pinchflat, :media_directory), "Season 1", "bar.mp4"])
{:ok, source_details_return_fixture(%{filename: filename})}
_url, opts, ot when ot == @metadata_ot ->
_url, opts, ot, _addl when ot == @metadata_ot ->
assert {:convert_thumbnails, "jpg"} in opts
{:ok, render_metadata(:channel_source_metadata)}
@@ -168,10 +168,10 @@ defmodule Pinchflat.Metadata.SourceMetadataStorageWorkerTest do
test "calls one set of yt-dlp metadata opts for channels" do
stub(YtDlpRunnerMock, :run, fn
_url, _opts, ot when ot == @source_details_ot ->
_url, _opts, ot, _addl when ot == @source_details_ot ->
{:ok, source_details_return_fixture()}
_url, opts, ot when ot == @metadata_ot ->
_url, opts, ot, _addl when ot == @metadata_ot ->
assert {:playlist_items, 0} in opts
assert :write_all_thumbnails in opts
@@ -186,10 +186,10 @@ defmodule Pinchflat.Metadata.SourceMetadataStorageWorkerTest do
test "calls another set of yt-dlp metadata opts for playlists" do
stub(YtDlpRunnerMock, :run, fn
_url, _opts, ot when ot == @source_details_ot ->
_url, _opts, ot, _addl when ot == @source_details_ot ->
{:ok, source_details_return_fixture()}
_url, opts, ot when ot == @metadata_ot ->
_url, opts, ot, _addl when ot == @metadata_ot ->
assert {:playlist_items, 1} in opts
assert :write_thumbnail in opts
@@ -204,12 +204,12 @@ defmodule Pinchflat.Metadata.SourceMetadataStorageWorkerTest do
test "does not store source images if the profile is not set to" do
stub(YtDlpRunnerMock, :run, fn
_url, _opts, ot when ot == @source_details_ot ->
_url, _opts, ot, _addl when ot == @source_details_ot ->
filename = Path.join([Application.get_env(:pinchflat, :media_directory), "Season 1", "bar.mp4"])
{:ok, source_details_return_fixture(%{filename: filename})}
_url, _opts, ot when ot == @metadata_ot ->
_url, _opts, ot, _addl when ot == @metadata_ot ->
{:ok, render_metadata(:channel_source_metadata)}
end)
@@ -226,12 +226,12 @@ defmodule Pinchflat.Metadata.SourceMetadataStorageWorkerTest do
test "does not store source images if the series directory cannot be determined" do
stub(YtDlpRunnerMock, :run, fn
_url, _opts, ot when ot == @source_details_ot ->
_url, _opts, ot, _addl when ot == @source_details_ot ->
filename = Path.join([Application.get_env(:pinchflat, :media_directory), "foo", "bar.mp4"])
{:ok, source_details_return_fixture(%{filename: filename})}
_url, _opts, ot when ot == @metadata_ot ->
_url, _opts, ot, _addl when ot == @metadata_ot ->
{:ok, render_metadata(:channel_source_metadata)}
end)
@@ -245,17 +245,49 @@ defmodule Pinchflat.Metadata.SourceMetadataStorageWorkerTest do
refute source.poster_filepath
refute source.banner_filepath
end
test "sets use_cookies if the source uses cookies" do
expect(YtDlpRunnerMock, :run, 2, fn
_url, _opts, ot, _addl when ot == @source_details_ot ->
{:ok, source_details_return_fixture()}
_url, _opts, ot, addl when ot == @metadata_ot ->
assert {:use_cookies, true} in addl
{:ok, render_metadata(:channel_source_metadata)}
end)
profile = media_profile_fixture(%{download_source_images: true})
source = source_fixture(media_profile_id: profile.id, use_cookies: true)
perform_job(SourceMetadataStorageWorker, %{id: source.id})
end
test "does not set use_cookies if the source does not use cookies" do
expect(YtDlpRunnerMock, :run, 2, fn
_url, _opts, ot, _addl when ot == @source_details_ot ->
{:ok, source_details_return_fixture()}
_url, _opts, ot, addl when ot == @metadata_ot ->
assert {:use_cookies, false} in addl
{:ok, render_metadata(:channel_source_metadata)}
end)
profile = media_profile_fixture(%{download_source_images: true})
source = source_fixture(media_profile_id: profile.id, use_cookies: false)
perform_job(SourceMetadataStorageWorker, %{id: source.id})
end
end
describe "perform/1 when determining the series_directory" do
test "sets the series directory based on the returned media filepath" do
stub(YtDlpRunnerMock, :run, fn
_url, _opts, ot when ot == @source_details_ot ->
_url, _opts, ot, _addl when ot == @source_details_ot ->
filename = Path.join([Application.get_env(:pinchflat, :media_directory), "Season 1", "bar.mp4"])
{:ok, source_details_return_fixture(%{filename: filename})}
_url, _opts, ot when ot == @metadata_ot ->
_url, _opts, ot, _addl when ot == @metadata_ot ->
{:ok, "{}"}
end)
@@ -268,12 +300,12 @@ defmodule Pinchflat.Metadata.SourceMetadataStorageWorkerTest do
test "does not set the series directory if it cannot be determined" do
stub(YtDlpRunnerMock, :run, fn
_url, _opts, ot when ot == @source_details_ot ->
_url, _opts, ot, _addl when ot == @source_details_ot ->
filename = Path.join([Application.get_env(:pinchflat, :media_directory), "foo", "bar.mp4"])
{:ok, source_details_return_fixture(%{filename: filename})}
_url, _opts, ot when ot == @metadata_ot ->
_url, _opts, ot, _addl when ot == @metadata_ot ->
{:ok, "{}"}
end)
@@ -283,17 +315,45 @@ defmodule Pinchflat.Metadata.SourceMetadataStorageWorkerTest do
refute source.series_directory
end
test "sets use_cookies if the source is set to use cookies" do
expect(YtDlpRunnerMock, :run, 2, fn
_url, _opts, ot, addl when ot == @source_details_ot ->
assert {:use_cookies, true} in addl
{:ok, source_details_return_fixture()}
_url, _opts, ot, _addl when ot == @metadata_ot ->
{:ok, "{}"}
end)
source = source_fixture(%{series_directory: nil, use_cookies: true})
perform_job(SourceMetadataStorageWorker, %{id: source.id})
end
test "does not set use_cookies if the source is not set to use cookies" do
expect(YtDlpRunnerMock, :run, 2, fn
_url, _opts, ot, addl when ot == @source_details_ot ->
assert {:use_cookies, false} in addl
{:ok, source_details_return_fixture()}
_url, _opts, ot, _addl when ot == @metadata_ot ->
{:ok, "{}"}
end)
source = source_fixture(%{series_directory: nil, use_cookies: false})
perform_job(SourceMetadataStorageWorker, %{id: source.id})
end
end
describe "perform/1 when storing the series NFO" do
test "stores the NFO if specified" do
stub(YtDlpRunnerMock, :run, fn
_url, _opts, ot when ot == @source_details_ot ->
_url, _opts, ot, _addl when ot == @source_details_ot ->
filename = Path.join([Application.get_env(:pinchflat, :media_directory), "Season 1", "bar.mp4"])
{:ok, source_details_return_fixture(%{filename: filename})}
_url, _opts, ot when ot == @metadata_ot ->
_url, _opts, ot, _addl when ot == @metadata_ot ->
{:ok, "{}"}
end)
@@ -311,12 +371,12 @@ defmodule Pinchflat.Metadata.SourceMetadataStorageWorkerTest do
test "does not store the NFO if not specified" do
stub(YtDlpRunnerMock, :run, fn
_url, _opts, ot when ot == @source_details_ot ->
_url, _opts, ot, _addl when ot == @source_details_ot ->
filename = Path.join([Application.get_env(:pinchflat, :media_directory), "Season 1", "bar.mp4"])
{:ok, source_details_return_fixture(%{filename: filename})}
_url, _opts, ot when ot == @metadata_ot ->
_url, _opts, ot, _addl when ot == @metadata_ot ->
{:ok, "{}"}
end)
@@ -330,12 +390,12 @@ defmodule Pinchflat.Metadata.SourceMetadataStorageWorkerTest do
test "does not store the NFO if the series directory cannot be determined" do
stub(YtDlpRunnerMock, :run, fn
_url, _opts, ot when ot == @source_details_ot ->
_url, _opts, ot, _addl when ot == @source_details_ot ->
filename = Path.join([Application.get_env(:pinchflat, :media_directory), "foo", "bar.mp4"])
{:ok, source_details_return_fixture(%{filename: filename})}
_url, _opts, ot when ot == @metadata_ot ->
_url, _opts, ot, _addl when ot == @metadata_ot ->
{:ok, "{}"}
end)
@@ -0,0 +1,57 @@
defmodule Pinchflat.Profiles.MediaProfileDeletionWorkerTest do
use Pinchflat.DataCase
import Pinchflat.MediaFixtures
import Pinchflat.SourcesFixtures
import Pinchflat.ProfilesFixtures
alias Pinchflat.Profiles.MediaProfileDeletionWorker
setup do
stub(UserScriptRunnerMock, :run, fn _event_type, _data -> :ok end)
{:ok, %{profile: media_profile_fixture()}}
end
describe "kickoff/3" do
test "starts the worker", %{profile: profile} do
assert [] = all_enqueued(worker: MediaProfileDeletionWorker)
assert {:ok, _} = MediaProfileDeletionWorker.kickoff(profile)
assert [_] = all_enqueued(worker: MediaProfileDeletionWorker)
end
test "can be called with additional job arguments", %{profile: profile} do
job_args = %{"delete_files" => true}
assert {:ok, _} = MediaProfileDeletionWorker.kickoff(profile, job_args)
assert_enqueued(worker: MediaProfileDeletionWorker, args: %{"id" => profile.id, "delete_files" => true})
end
end
describe "perform/1" do
test "deletes the profile, sources, and media but leaves the files", %{profile: profile} do
source = source_fixture(%{media_profile_id: profile.id})
media_item = media_item_with_attachments(%{source_id: source.id})
perform_job(MediaProfileDeletionWorker, %{"id" => profile.id})
assert_raise Ecto.NoResultsError, fn -> Repo.reload!(profile) end
assert_raise Ecto.NoResultsError, fn -> Repo.reload!(source) end
assert_raise Ecto.NoResultsError, fn -> Repo.reload!(media_item) end
assert File.exists?(media_item.media_filepath)
end
test "deletes the profile, sources, and media and files if specified", %{profile: profile} do
source = source_fixture(%{media_profile_id: profile.id})
media_item = media_item_with_attachments(%{source_id: source.id})
perform_job(MediaProfileDeletionWorker, %{"id" => profile.id, "delete_files" => true})
assert_raise Ecto.NoResultsError, fn -> Repo.reload!(profile) end
assert_raise Ecto.NoResultsError, fn -> Repo.reload!(source) end
assert_raise Ecto.NoResultsError, fn -> Repo.reload!(media_item) end
refute File.exists?(media_item.media_filepath)
end
end
end

Some files were not shown because too many files have changed in this diff Show More