Compare commits

..

11 Commits

Author SHA1 Message Date
Kieran Eglin db859411dd Bumped version 2024-04-11 15:21:02 -07:00
Kieran 0aa4b5b119 [Enhancement] Improve support for 4k videos with Plex (#181)
* Added WIP 4k MP4 fix [skip ci]

* Added tests for new remux options
2024-04-11 15:17:17 -07:00
Kieran 17f4bd70f7 [Housekeeping] Close system port when jobs are cancelled (#182)
* Created a test setup that works

* Refactored test setup into real-world fixes
2024-04-11 15:04:19 -07:00
Kieran 2c717e8e7e [Enhancement] Allow overriding output templates on a per-source basis (#179)
* Added output path override to table and download option builder

* Added output template override to UI
2024-04-11 10:46:19 -07:00
Kieran Eglin 3acdb5ccba Bumped version 2024-04-10 16:59:01 -07:00
Kieran Eglin 0ba74264c4 Fixed bug with redownloading not forcing download of the video 2024-04-10 16:43:15 -07:00
Kieran Eglin 72c82ee72f Updated README blurb on WAL issues 2024-04-10 16:01:50 -07:00
Kieran Eglin 3018a9c46b Bumped version 2024-04-10 14:35:45 -07:00
Kieran 314118341e [Enhancement] Allow custom yt-dlp options (#176)
* Added option for yt-dlp config file usage

* renamed yt-dlp config file

* refactored to use a precedence-based approach

* Updated README
2024-04-10 13:17:22 -07:00
Kieran Eglin b4438c1ba5 Renamed FS Helpers module to FS Utils 2024-04-10 09:02:26 -07:00
Kieran 725edc3fd8 [Enhancement] Redownload new media after a delay (#173)
* Added redownload-related columns

* Added methods for fetching re-downloadable media items

* Filled out redownload worker + tests

* Added redownload worker to config.exs cron

* Added to UI and README
2024-04-10 08:54:45 -07:00
45 changed files with 884 additions and 232 deletions
+5 -2
View File
@@ -49,11 +49,14 @@ If it doesn't work for your use case, please make a feature request! You can als
- Supports downloading audio content - Supports downloading audio content
- Custom rules for handling YouTube Shorts and livestreams - Custom rules for handling YouTube Shorts and livestreams
- Apprise support for notifications - Apprise support for notifications
- Allows automatically redownloading new media after a set period
- This can help improve the download quality of new content or improve SponsorBlock tags
- Optionally automatically delete old content ([docs](https://github.com/kieraneglin/pinchflat/wiki/Automatically-Delete-Media)) - Optionally automatically delete old content ([docs](https://github.com/kieraneglin/pinchflat/wiki/Automatically-Delete-Media))
- Advanced options like setting cutoff dates and filtering by title - Advanced options like setting cutoff dates and filtering by title
- Reliable hands-off operation - Reliable hands-off operation
- Can pass cookies to YouTube to download your private playlists ([docs](https://github.com/kieraneglin/pinchflat/wiki/YouTube-Cookies)) - Can pass cookies to YouTube to download your private playlists ([docs](https://github.com/kieraneglin/pinchflat/wiki/YouTube-Cookies))
- Sponsorblock integration - Sponsorblock integration
- \[Advanced\] allows custom `yt-dlp` options ([docs](https://github.com/kieraneglin/pinchflat/wiki/%5BAdvanced%5D-Custom-yt%E2%80%90dlp-options))
## Screenshots ## Screenshots
@@ -111,10 +114,10 @@ It's recommended to not run the container as root. Doing so can create permissio
### Advanced: storing Pinchflat config directory on a network share ### Advanced: storing Pinchflat config directory on a network share
README: This is currently in the testing phase and not a recommended option (yet). The implications of changing this setting isn't clear and this could, conceivably, result in data loss. Only change this setting if you know what you're doing, why this is important, and are okay with possible data loss or DB corruption. This may become the default in the future once it's been tested more thoroughly.
As pointed out in [#137](https://github.com/kieraneglin/pinchflat/issues/137), SQLite doesn't like being run in WAL mode on network shares. If you're running Pinchflat on a network share, you can disable WAL mode by setting the `JOURNAL_MODE` environment variable to `delete`. This will make Pinchflat run in rollback journal mode which is less performant but should work on network shares. As pointed out in [#137](https://github.com/kieraneglin/pinchflat/issues/137), SQLite doesn't like being run in WAL mode on network shares. If you're running Pinchflat on a network share, you can disable WAL mode by setting the `JOURNAL_MODE` environment variable to `delete`. This will make Pinchflat run in rollback journal mode which is less performant but should work on network shares.
Changing this setting from WAL to `delete` on an existing Pinchflat instance could, conceivably, result in data loss. Only change this setting if you know what you're doing, why this is important, and are okay with possible data loss or DB corruption. Backup your database first!
If you change this setting and it works well for you, please leave a comment on [#137](https://github.com/kieraneglin/pinchflat/issues/137)! Doubly so if it does _not_ work well. If you change this setting and it works well for you, please leave a comment on [#137](https://github.com/kieraneglin/pinchflat/issues/137)! Doubly so if it does _not_ work well.
## EFF donations ## EFF donations
+2 -1
View File
@@ -52,7 +52,8 @@ config :pinchflat, Oban,
{Oban.Plugins.Pruner, max_age: 30 * 24 * 60 * 60}, {Oban.Plugins.Pruner, max_age: 30 * 24 * 60 * 60},
{Oban.Plugins.Cron, {Oban.Plugins.Cron,
crontab: [ crontab: [
{"@daily", Pinchflat.Downloading.MediaRetentionWorker} {"0 1 * * *", Pinchflat.Downloading.MediaRetentionWorker},
{"0 2 * * *", Pinchflat.Downloading.MediaRedownloadWorker}
]} ]}
], ],
# TODO: consider making this an env var or something? # TODO: consider making this an env var or something?
+1 -1
View File
@@ -8,7 +8,7 @@ FROM ${DEV_IMAGE}
# Install debian packages # Install debian packages
RUN apt-get update -qq RUN apt-get update -qq
RUN apt-get install -y inotify-tools ffmpeg curl git openssh-client \ RUN apt-get install -y inotify-tools ffmpeg curl git openssh-client \
python3 python3-pip python3-setuptools python3-wheel python3-dev locales python3 python3-pip python3-setuptools python3-wheel python3-dev locales procps
# Install nodejs # Install nodejs
RUN curl -sL https://deb.nodesource.com/setup_20.x -o nodesource_setup.sh RUN curl -sL https://deb.nodesource.com/setup_20.x -o nodesource_setup.sh
+10 -6
View File
@@ -14,7 +14,7 @@ defmodule Pinchflat.Boot.PreJobStartupTasks do
alias Pinchflat.Repo alias Pinchflat.Repo
alias Pinchflat.Settings alias Pinchflat.Settings
alias Pinchflat.Filesystem.FilesystemHelpers alias Pinchflat.Utils.FilesystemUtils
def start_link(opts \\ []) do def start_link(opts \\ []) do
GenServer.start_link(__MODULE__, %{}, opts) GenServer.start_link(__MODULE__, %{}, opts)
@@ -32,7 +32,7 @@ defmodule Pinchflat.Boot.PreJobStartupTasks do
@impl true @impl true
def init(state) do def init(state) do
reset_executing_jobs() reset_executing_jobs()
create_blank_cookie_file() create_blank_yt_dlp_files()
apply_default_settings() apply_default_settings()
{:ok, state} {:ok, state}
@@ -50,15 +50,19 @@ defmodule Pinchflat.Boot.PreJobStartupTasks do
Logger.info("Reset #{count} executing jobs") Logger.info("Reset #{count} executing jobs")
end end
defp create_blank_cookie_file do defp create_blank_yt_dlp_files do
files = ["cookies.txt", "yt-dlp-configs/base-config.txt"]
base_dir = Application.get_env(:pinchflat, :extras_directory) base_dir = Application.get_env(:pinchflat, :extras_directory)
filepath = Path.join(base_dir, "cookies.txt")
Enum.each(files, fn file ->
filepath = Path.join(base_dir, file)
if !File.exists?(filepath) do if !File.exists?(filepath) do
Logger.info("yt-dlp cookie file does not exist - creating it") Logger.info("Creating blank file: #{filepath}")
FilesystemHelpers.write_p!(filepath, "") FilesystemUtils.write_p!(filepath, "")
end end
end)
end end
defp apply_default_settings do defp apply_default_settings do
@@ -3,6 +3,7 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilder do
Builds the options for yt-dlp to download media based on the given media profile. Builds the options for yt-dlp to download media based on the given media profile.
""" """
alias Pinchflat.Sources
alias Pinchflat.Sources.Source alias Pinchflat.Sources.Source
alias Pinchflat.Media.MediaItem alias Pinchflat.Media.MediaItem
alias Pinchflat.Downloading.OutputPathBuilder alias Pinchflat.Downloading.OutputPathBuilder
@@ -10,8 +11,7 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilder do
@doc """ @doc """
Builds the options for yt-dlp to download media based on the given media's profile. Builds the options for yt-dlp to download media based on the given media's profile.
IDEA: consider adding the ability to pass in a second argument to override Returns {:ok, [Keyword.t()]}
these options
""" """
def build(%MediaItem{} = media_item_with_preloads) do def build(%MediaItem{} = media_item_with_preloads) do
media_profile = media_item_with_preloads.source.media_profile media_profile = media_item_with_preloads.source.media_profile
@@ -23,25 +23,27 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilder do
metadata_options(media_profile) ++ metadata_options(media_profile) ++
quality_options(media_profile) ++ quality_options(media_profile) ++
sponsorblock_options(media_profile) ++ sponsorblock_options(media_profile) ++
output_options(media_item_with_preloads) output_options(media_item_with_preloads) ++
config_file_options(media_item_with_preloads)
{:ok, built_options} {:ok, built_options}
end end
@doc """ @doc """
Builds the output path for yt-dlp to download media based on the given source's Builds the output path for yt-dlp to download media based on the given source's
media profile. media profile. Uses the source's override output path template if it exists.
Returns binary() Returns binary()
""" """
def build_output_path_for(%Source{} = source_with_preloads) do def build_output_path_for(%Source{} = source_with_preloads) do
output_path_template = source_with_preloads.media_profile.output_path_template output_path_template = Sources.output_path_template(source_with_preloads)
build_output_path(output_path_template, source_with_preloads) build_output_path(output_path_template, source_with_preloads)
end end
defp default_options do defp default_options do
[:no_progress, :windows_filenames] # Add force-overwrites to make sure redownloading works
[:no_progress, :windows_filenames, :force_overwrites]
end end
defp subtitle_options(media_profile) do defp subtitle_options(media_profile) do
@@ -104,16 +106,18 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilder do
end end
defp quality_options(media_profile) do defp quality_options(media_profile) do
video_codec_options = "+codec:avc:m4a" video_codec_option = fn res ->
[format_sort: "res:#{res},+codec:avc:m4a", remux_video: "mp4"]
end
case media_profile.preferred_resolution do case media_profile.preferred_resolution do
# Also be aware that :audio disabled all embedding options for subtitles # Also be aware that :audio disabled all embedding options for subtitles
:audio -> [:extract_audio, format: "bestaudio[ext=m4a]"] :audio -> [:extract_audio, format: "bestaudio[ext=m4a]"]
:"360p" -> [format_sort: "res:360,#{video_codec_options}"] :"360p" -> video_codec_option.("360")
:"480p" -> [format_sort: "res:480,#{video_codec_options}"] :"480p" -> video_codec_option.("480")
:"720p" -> [format_sort: "res:720,#{video_codec_options}"] :"720p" -> video_codec_option.("720")
:"1080p" -> [format_sort: "res:1080,#{video_codec_options}"] :"1080p" -> video_codec_option.("1080")
:"2160p" -> [format_sort: "res:2160,#{video_codec_options}"] :"2160p" -> video_codec_option.("2160")
end end
end end
@@ -128,6 +132,35 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilder do
end end
end end
# This is put here instead of the CommandRunner module because it should only
# be applied to downloading - if it were in CommandRunner it would apply to
# all yt-dlp commands (like indexing)
defp config_file_options(media_item) do
base_dir = Path.join(Application.get_env(:pinchflat, :extras_directory), "yt-dlp-configs")
# Ordered by priority - the first file has the highest priority
filenames = [
"media-item-#{media_item.id}-config.txt",
"source-#{media_item.source_id}-config.txt",
"media-profile-#{media_item.source.media_profile_id}-config.txt",
"base-config.txt"
]
config_filepaths =
Enum.reduce(filenames, [], fn filename, acc ->
filepath = Path.join(base_dir, filename)
case File.read(filepath) do
{:ok, file_data} ->
if String.trim(file_data) != "", do: [filepath | acc], else: acc
{:error, _} ->
acc
end
end)
Enum.map(config_filepaths, fn filepath -> {:config_locations, filepath} end)
end
defp output_options(media_item_with_preloads) do defp output_options(media_item_with_preloads) do
[ [
output: build_output_path_for(media_item_with_preloads.source) output: build_output_path_for(media_item_with_preloads.source)
@@ -154,7 +187,7 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilder do
# It's dependent on the output_path_template being a string ending `.{{ ext }}` # It's dependent on the output_path_template being a string ending `.{{ ext }}`
# (or equivalent), but that's validated by the MediaProfile schema. # (or equivalent), but that's validated by the MediaProfile schema.
defp determine_thumbnail_location(media_item_with_preloads) do defp determine_thumbnail_location(media_item_with_preloads) do
output_path_template = media_item_with_preloads.source.media_profile.output_path_template output_path_template = Sources.output_path_template(media_item_with_preloads.source)
output_path_template output_path_template
|> String.split(~r{\.}, include_captures: true) |> String.split(~r{\.}, include_captures: true)
@@ -35,14 +35,17 @@ defmodule Pinchflat.Downloading.MediaDownloadWorker do
""" """
@impl Oban.Worker @impl Oban.Worker
def perform(%Oban.Job{args: %{"id" => media_item_id} = args}) do def perform(%Oban.Job{args: %{"id" => media_item_id} = args}) do
should_force = Map.get(args, "force", false)
is_redownload = Map.get(args, "redownload?", false)
media_item = media_item =
media_item_id media_item_id
|> Media.get_media_item!() |> Media.get_media_item!()
|> Repo.preload(:source) |> Repo.preload(:source)
# If the source or media item is set to not download media, perform a no-op unless forced # If the source or media item is set to not download media, perform a no-op unless forced
if (media_item.source.download_media && !media_item.prevent_download) || args["force"] do if (media_item.source.download_media && !media_item.prevent_download) || should_force do
download_media_and_schedule_jobs(media_item) download_media_and_schedule_jobs(media_item, is_redownload)
else else
:ok :ok
end end
@@ -51,10 +54,13 @@ defmodule Pinchflat.Downloading.MediaDownloadWorker do
Ecto.StaleEntryError -> Logger.info("#{__MODULE__} discarded: media item #{media_item_id} stale") Ecto.StaleEntryError -> Logger.info("#{__MODULE__} discarded: media item #{media_item_id} stale")
end end
defp download_media_and_schedule_jobs(media_item) do defp download_media_and_schedule_jobs(media_item, is_redownload) do
case MediaDownloader.download_for_media_item(media_item) do case MediaDownloader.download_for_media_item(media_item) do
{:ok, updated_media_item} -> {:ok, updated_media_item} ->
compute_and_save_media_filesize(updated_media_item) Media.update_media_item(updated_media_item, %{
media_size_bytes: compute_media_filesize(updated_media_item),
media_redownloaded_at: get_redownloaded_at(is_redownload)
})
{:ok, updated_media_item} {:ok, updated_media_item}
@@ -66,13 +72,21 @@ defmodule Pinchflat.Downloading.MediaDownloadWorker do
end end
end end
defp compute_and_save_media_filesize(media_item) do defp compute_media_filesize(media_item) do
case File.stat(media_item.media_filepath) do case File.stat(media_item.media_filepath) do
{:ok, %{size: size}} -> {:ok, %{size: size}} ->
Media.update_media_item(media_item, %{media_size_bytes: size}) size
_ -> _ ->
:ok nil
end
end
defp get_redownloaded_at(is_redownload) do
if is_redownload do
DateTime.utc_now()
else
nil
end end
end end
end end
@@ -13,7 +13,7 @@ defmodule Pinchflat.Downloading.MediaDownloader do
alias Pinchflat.Metadata.NfoBuilder alias Pinchflat.Metadata.NfoBuilder
alias Pinchflat.Metadata.MetadataParser alias Pinchflat.Metadata.MetadataParser
alias Pinchflat.Metadata.MetadataFileHelpers alias Pinchflat.Metadata.MetadataFileHelpers
alias Pinchflat.Filesystem.FilesystemHelpers alias Pinchflat.Utils.FilesystemUtils
alias Pinchflat.Downloading.DownloadOptionBuilder alias Pinchflat.Downloading.DownloadOptionBuilder
alias Pinchflat.YtDlp.Media, as: YtDlpMedia alias Pinchflat.YtDlp.Media, as: YtDlpMedia
@@ -30,7 +30,7 @@ defmodule Pinchflat.Downloading.MediaDownloader do
Returns {:ok, %MediaItem{}} | {:error, any, ...any} Returns {:ok, %MediaItem{}} | {:error, any, ...any}
""" """
def download_for_media_item(%MediaItem{} = media_item) do def download_for_media_item(%MediaItem{} = media_item) do
output_filepath = FilesystemHelpers.generate_metadata_tmpfile(:json) output_filepath = FilesystemUtils.generate_metadata_tmpfile(:json)
media_with_preloads = Repo.preload(media_item, [:metadata, source: :media_profile]) media_with_preloads = Repo.preload(media_item, [:metadata, source: :media_profile])
case download_with_options(media_item.original_url, media_with_preloads, output_filepath) do case download_with_options(media_item.original_url, media_with_preloads, output_filepath) do
@@ -0,0 +1,31 @@
defmodule Pinchflat.Downloading.MediaRedownloadWorker do
@moduledoc false
use Oban.Worker,
queue: :media_fetching,
unique: [period: :infinity, states: [:available, :scheduled, :retryable, :executing]],
tags: ["media_item", "media_fetching"]
require Logger
alias Pinchflat.Media
alias Pinchflat.Downloading.MediaDownloadWorker
@doc """
Redownloads media items that are eligible for redownload.
This worker is scheduled to run daily via the Oban Cron plugin
and it should run _after_ the retention worker.
Returns :ok
"""
@impl Oban.Worker
def perform(%Oban.Job{}) do
redownloadable_media = Media.list_redownloadable_media_items()
Logger.info("Redownloading #{length(redownloadable_media)} media items")
Enum.each(redownloadable_media, fn media_item ->
MediaDownloadWorker.kickoff_with_task(media_item, %{redownload?: true})
end)
end
end
+32 -8
View File
@@ -11,7 +11,7 @@ defmodule Pinchflat.Media do
alias Pinchflat.Media.MediaItem alias Pinchflat.Media.MediaItem
alias Pinchflat.Media.MediaQuery alias Pinchflat.Media.MediaQuery
alias Pinchflat.Metadata.MediaMetadata alias Pinchflat.Metadata.MediaMetadata
alias Pinchflat.Filesystem.FilesystemHelpers alias Pinchflat.Utils.FilesystemUtils
@doc """ @doc """
Returns the list of media_items. Returns the list of media_items.
@@ -31,15 +31,39 @@ defmodule Pinchflat.Media do
def list_cullable_media_items do def list_cullable_media_items do
MediaQuery.new() MediaQuery.new()
|> MediaQuery.with_media_filepath() |> MediaQuery.with_media_filepath()
|> MediaQuery.with_passed_retention_period() |> MediaQuery.where_past_retention_period()
|> MediaQuery.with_no_culling_prevention() |> MediaQuery.where_culling_not_prevented()
|> Repo.all()
end
@doc """
Returns a list of media_items that are redownloadable based on the redownload delay
of the media_profile their source belongs to.
The logic is that a media_item is past_redownload_delay if the media_item's
upload_date is at least redownload_delay_days ago AND
`media_downloaded_at` - `redownload_delay_days` is before the media_item's `upload_date`.
This logic grabs media that we've recently downloaded AND is recently uploaded, but
doesn't grab media that we've recently downloaded and was uploaded a long time ago.
This also makes things work as expected when downloading media from a source for the
first time.
Returns [%MediaItem{}, ...]
"""
def list_redownloadable_media_items do
MediaQuery.new()
|> MediaQuery.with_media_downloaded_at()
|> MediaQuery.where_download_not_prevented()
|> MediaQuery.where_not_culled()
|> MediaQuery.where_media_not_redownloaded()
|> MediaQuery.where_past_redownload_delay()
|> Repo.all() |> Repo.all()
end end
@doc """ @doc """
Returns a list of pending media_items for a given source, where Returns a list of pending media_items for a given source, where
pending means the `media_filepath` is `nil` AND the media_item pending means the `media_filepath` is `nil` AND the media_item
matches satisfies `MediaQuery.with_media_pending_download`. You matches satisfies `MediaQuery.where_pending_download`. You
should really check out that function if you need to know more should really check out that function if you need to know more
because it has a lot going on. because it has a lot going on.
@@ -48,7 +72,7 @@ defmodule Pinchflat.Media do
def list_pending_media_items_for(%Source{} = source) do def list_pending_media_items_for(%Source{} = source) do
MediaQuery.new() MediaQuery.new()
|> MediaQuery.for_source(source) |> MediaQuery.for_source(source)
|> MediaQuery.with_media_pending_download() |> MediaQuery.where_pending_download()
|> Repo.all() |> Repo.all()
end end
@@ -66,7 +90,7 @@ defmodule Pinchflat.Media do
MediaQuery.new() MediaQuery.new()
|> MediaQuery.with_id(media_item.id) |> MediaQuery.with_id(media_item.id)
|> MediaQuery.with_media_pending_download() |> MediaQuery.where_pending_download()
|> Repo.exists?() |> Repo.exists?()
end end
@@ -199,7 +223,7 @@ defmodule Pinchflat.Media do
end) end)
|> List.flatten() |> List.flatten()
|> Enum.filter(&is_binary/1) |> Enum.filter(&is_binary/1)
|> Enum.each(&FilesystemHelpers.delete_file_and_remove_empty_directories/1) |> Enum.each(&FilesystemUtils.delete_file_and_remove_empty_directories/1)
{:ok, media_item} {:ok, media_item}
end end
@@ -211,6 +235,6 @@ defmodule Pinchflat.Media do
MediaMetadata.filepath_attributes() MediaMetadata.filepath_attributes()
|> Enum.map(fn field -> mapped_struct[field] end) |> Enum.map(fn field -> mapped_struct[field] end)
|> Enum.filter(&is_binary/1) |> Enum.filter(&is_binary/1)
|> Enum.each(&FilesystemHelpers.delete_file_and_remove_empty_directories/1) |> Enum.each(&FilesystemUtils.delete_file_and_remove_empty_directories/1)
end end
end end
+3 -1
View File
@@ -34,7 +34,8 @@ defmodule Pinchflat.Media.MediaItem do
# These are user or system controlled fields # These are user or system controlled fields
:prevent_download, :prevent_download,
:prevent_culling, :prevent_culling,
:culled_at :culled_at,
:media_redownloaded_at
] ]
# Pretty much all the fields captured at index are required. # Pretty much all the fields captured at index are required.
@required_fields ~w( @required_fields ~w(
@@ -61,6 +62,7 @@ defmodule Pinchflat.Media.MediaItem do
field :livestream, :boolean, default: false field :livestream, :boolean, default: false
field :short_form_content, :boolean, default: false field :short_form_content, :boolean, default: false
field :media_downloaded_at, :utc_datetime field :media_downloaded_at, :utc_datetime
field :media_redownloaded_at, :utc_datetime
field :upload_date, :date field :upload_date, :date
field :duration_seconds, :integer field :duration_seconds, :integer
+38 -12
View File
@@ -15,7 +15,7 @@ defmodule Pinchflat.Media.MediaQuery do
# Prefixes: # Prefixes:
# - for_* - belonging to a certain record # - for_* - belonging to a certain record
# - join_* - for joining on a certain record # - join_* - for joining on a certain record
# - with_* - for filtering based on full, concrete attributes # - with_*, where_* - for filtering based on full, concrete attributes
# - matching_* - for filtering based on partial attributes (e.g. LIKE, regex, full-text search) # - matching_* - for filtering based on partial attributes (e.g. LIKE, regex, full-text search)
# #
# Suffixes: # Suffixes:
@@ -33,24 +33,46 @@ defmodule Pinchflat.Media.MediaQuery do
from(mi in query, join: s in assoc(mi, :source), as: :sources) from(mi in query, join: s in assoc(mi, :source), as: :sources)
end end
def with_passed_retention_period(query) do def where_past_retention_period(query) do
query query
|> require_assoc(:source) |> require_assoc(:source)
|> where( |> where(
[mi, source], [mi, source],
fragment( fragment("""
"IFNULL(?, 0) > 0 AND DATETIME('now', '-' || ? || ' day') > ?", IFNULL(retention_period_days, 0) > 0 AND
source.retention_period_days, DATETIME('now', '-' || retention_period_days || ' day') > media_downloaded_at
source.retention_period_days, """)
mi.media_downloaded_at
)
) )
end end
def with_no_culling_prevention(query) do def where_past_redownload_delay(query) do
query
|> require_assoc(:source)
|> require_assoc(:media_profile)
|> where(
[_mi, _source, _media_profile],
# Returns media items where the upload_date is at least redownload_delay_days ago AND
# downloaded_at minus the redownload_delay_days is before the upload date
fragment("""
IFNULL(redownload_delay_days, 0) > 0 AND
DATETIME('now', '-' || redownload_delay_days || ' day') > upload_date AND
DATETIME(media_downloaded_at, '-' || redownload_delay_days || ' day') < upload_date
""")
)
end
def where_culling_not_prevented(query) do
where(query, [mi], mi.prevent_culling == false) where(query, [mi], mi.prevent_culling == false)
end end
def where_not_culled(query) do
where(query, [mi], is_nil(mi.culled_at))
end
def where_media_not_redownloaded(query) do
where(query, [mi], is_nil(mi.media_redownloaded_at))
end
def with_id(query, id) do def with_id(query, id) do
where(query, [mi], mi.id == ^id) where(query, [mi], mi.id == ^id)
end end
@@ -59,6 +81,10 @@ defmodule Pinchflat.Media.MediaQuery do
where(query, [mi], mi.media_id in ^media_ids) where(query, [mi], mi.media_id in ^media_ids)
end end
def with_media_downloaded_at(query) do
where(query, [mi], not is_nil(mi.media_downloaded_at))
end
def with_media_filepath(query) do def with_media_filepath(query) do
where(query, [mi], not is_nil(mi.media_filepath)) where(query, [mi], not is_nil(mi.media_filepath))
end end
@@ -73,7 +99,7 @@ defmodule Pinchflat.Media.MediaQuery do
|> where([mi, source], is_nil(source.download_cutoff_date) or mi.upload_date >= source.download_cutoff_date) |> where([mi, source], is_nil(source.download_cutoff_date) or mi.upload_date >= source.download_cutoff_date)
end end
def with_no_prevented_download(query) do def where_download_not_prevented(query) do
where(query, [mi], mi.prevent_download == false) where(query, [mi], mi.prevent_download == false)
end end
@@ -129,9 +155,9 @@ defmodule Pinchflat.Media.MediaQuery do
) )
end end
def with_media_pending_download(query) do def where_pending_download(query) do
query query
|> with_no_prevented_download() |> where_download_not_prevented()
|> with_no_media_filepath() |> with_no_media_filepath()
|> with_upload_date_after_source_cutoff() |> with_upload_date_after_source_cutoff()
|> with_format_matching_profile_preference() |> with_format_matching_profile_preference()
@@ -9,7 +9,7 @@ defmodule Pinchflat.Metadata.MetadataFileHelpers do
needed needed
""" """
alias Pinchflat.Filesystem.FilesystemHelpers alias Pinchflat.Utils.FilesystemUtils
@doc """ @doc """
Returns the directory where metadata for a database record should be stored. Returns the directory where metadata for a database record should be stored.
@@ -36,7 +36,7 @@ defmodule Pinchflat.Metadata.MetadataFileHelpers do
filepath = generate_filepath_for(database_record, "metadata.json.gz") filepath = generate_filepath_for(database_record, "metadata.json.gz")
{:ok, json} = Phoenix.json_library().encode(metadata_map) {:ok, json} = Phoenix.json_library().encode(metadata_map)
:ok = FilesystemHelpers.write_p!(filepath, json, [:compressed]) :ok = FilesystemUtils.write_p!(filepath, json, [:compressed])
filepath filepath
end end
@@ -62,7 +62,7 @@ defmodule Pinchflat.Metadata.MetadataFileHelpers do
filepath = generate_filepath_for(database_record, Path.basename(thumbnail_url)) filepath = generate_filepath_for(database_record, Path.basename(thumbnail_url))
thumbnail_blob = fetch_thumbnail_from_url(thumbnail_url) thumbnail_blob = fetch_thumbnail_from_url(thumbnail_url)
:ok = FilesystemHelpers.write_p!(filepath, thumbnail_blob) :ok = FilesystemUtils.write_p!(filepath, thumbnail_blob)
filepath filepath
end end
+3 -3
View File
@@ -7,7 +7,7 @@ defmodule Pinchflat.Metadata.NfoBuilder do
import Pinchflat.Utils.XmlUtils, only: [safe: 1] import Pinchflat.Utils.XmlUtils, only: [safe: 1]
alias Pinchflat.Metadata.MetadataFileHelpers alias Pinchflat.Metadata.MetadataFileHelpers
alias Pinchflat.Filesystem.FilesystemHelpers alias Pinchflat.Utils.FilesystemUtils
@doc """ @doc """
Builds an NFO file for a media item (read: single "episode") and Builds an NFO file for a media item (read: single "episode") and
@@ -18,7 +18,7 @@ defmodule Pinchflat.Metadata.NfoBuilder do
def build_and_store_for_media_item(filepath, metadata) do def build_and_store_for_media_item(filepath, metadata) do
nfo = build_for_media_item(metadata) nfo = build_for_media_item(metadata)
FilesystemHelpers.write_p!(filepath, nfo) FilesystemUtils.write_p!(filepath, nfo)
filepath filepath
end end
@@ -32,7 +32,7 @@ defmodule Pinchflat.Metadata.NfoBuilder do
def build_and_store_for_source(filepath, metadata) do def build_and_store_for_source(filepath, metadata) do
nfo = build_for_source(metadata) nfo = build_for_source(metadata)
FilesystemHelpers.write_p!(filepath, nfo) FilesystemUtils.write_p!(filepath, nfo)
filepath filepath
end end
@@ -2,7 +2,7 @@ defmodule Pinchflat.Metadata.SourceImageParser do
@moduledoc """ @moduledoc """
Functions for parsing and storing source images. Functions for parsing and storing source images.
""" """
alias Pinchflat.Filesystem.FilesystemHelpers alias Pinchflat.Utils.FilesystemUtils
@doc """ @doc """
Given a base directory and source metadata, look for the appropriate images Given a base directory and source metadata, look for the appropriate images
@@ -62,7 +62,7 @@ defmodule Pinchflat.Metadata.SourceImageParser do
extension = Path.extname(tmp_filepath) extension = Path.extname(tmp_filepath)
final_filepath = Path.join([base_directory, "#{filename}#{extension}"]) final_filepath = Path.join([base_directory, "#{filename}#{extension}"])
FilesystemHelpers.cp_p!(tmp_filepath, final_filepath) FilesystemUtils.cp_p!(tmp_filepath, final_filepath)
{source_attr_name, final_filepath} {source_attr_name, final_filepath}
end end
@@ -45,7 +45,7 @@ defmodule Pinchflat.Notifications.CommandRunner do
""" """
@impl AppriseCommandRunner @impl AppriseCommandRunner
def version do def version do
case System.cmd(backend_executable(), ["--version"]) do case CliUtils.wrap_cmd(backend_executable(), ["--version"]) do
{output, 0} -> {output, 0} ->
output output
|> String.split(~r{\r?\n}) |> String.split(~r{\r?\n})
@@ -59,7 +59,7 @@ defmodule Pinchflat.Notifications.SourceNotifications do
defp pending_media_item_count(source) do defp pending_media_item_count(source) do
MediaQuery.new() MediaQuery.new()
|> MediaQuery.for_source(source) |> MediaQuery.for_source(source)
|> MediaQuery.with_media_pending_download() |> MediaQuery.where_pending_download()
|> Repo.aggregate(:count) |> Repo.aggregate(:count)
end end
+5 -2
View File
@@ -26,12 +26,14 @@ defmodule Pinchflat.Profiles.MediaProfile do
shorts_behaviour shorts_behaviour
livestream_behaviour livestream_behaviour
preferred_resolution preferred_resolution
redownload_delay_days
)a )a
@required_fields ~w(name output_path_template)a @required_fields ~w(name output_path_template)a
schema "media_profiles" do schema "media_profiles" do
field :name, :string field :name, :string
field :redownload_delay_days, :integer
field :output_path_template, :string, field :output_path_template, :string,
default: "/{{ source_custom_name }}/{{ upload_yyyy_mm_dd }} {{ title }}/{{ title }} [{{ id }}].{{ ext }}" default: "/{{ source_custom_name }}/{{ upload_yyyy_mm_dd }} {{ title }}/{{ title }} [{{ id }}].{{ ext }}"
@@ -60,7 +62,6 @@ defmodule Pinchflat.Profiles.MediaProfile do
# See `build_format_clauses` in the Media context for more. # See `build_format_clauses` in the Media context for more.
field :shorts_behaviour, Ecto.Enum, values: ~w(include exclude only)a, default: :include field :shorts_behaviour, Ecto.Enum, values: ~w(include exclude only)a, default: :include
field :livestream_behaviour, Ecto.Enum, values: ~w(include exclude only)a, default: :include field :livestream_behaviour, Ecto.Enum, values: ~w(include exclude only)a, default: :include
field :preferred_resolution, Ecto.Enum, values: ~w(2160p 1080p 720p 480p 360p audio)a, default: :"1080p" field :preferred_resolution, Ecto.Enum, values: ~w(2160p 1080p 720p 480p 360p audio)a, default: :"1080p"
has_many :sources, Source has_many :sources, Source
@@ -75,10 +76,12 @@ defmodule Pinchflat.Profiles.MediaProfile do
|> validate_required(@required_fields) |> validate_required(@required_fields)
# Ensures it ends with `.{{ ext }}` or `.%(ext)s` or similar (with a little wiggle room) # Ensures it ends with `.{{ ext }}` or `.%(ext)s` or similar (with a little wiggle room)
|> validate_format(:output_path_template, ext_regex(), message: "must end with .{{ ext }}") |> validate_format(:output_path_template, ext_regex(), message: "must end with .{{ ext }}")
|> validate_number(:redownload_delay_days, greater_than_or_equal_to: 0)
|> unique_constraint(:name) |> unique_constraint(:name)
end end
defp ext_regex do @doc false
def ext_regex do
~r/\.({{ ?ext ?}}|%\( ?ext ?\)[sS])$/ ~r/\.({{ ?ext ?}}|%\( ?ext ?\)[sS])$/
end end
end end
+2 -2
View File
@@ -7,7 +7,7 @@ defmodule Pinchflat.Release do
require Logger require Logger
alias Pinchflat.Filesystem.FilesystemHelpers alias Pinchflat.Utils.FilesystemUtils
def migrate do def migrate do
load_app() load_app()
@@ -39,7 +39,7 @@ defmodule Pinchflat.Release do
Logger.info("Checking permissions for #{dir}") Logger.info("Checking permissions for #{dir}")
filepath = Path.join([dir, ".keep"]) filepath = Path.join([dir, ".keep"])
case FilesystemHelpers.write_p(filepath, "") do case FilesystemUtils.write_p(filepath, "") do
:ok -> :ok ->
Logger.info("Permissions OK") Logger.info("Permissions OK")
+4
View File
@@ -32,6 +32,7 @@ defmodule Pinchflat.Sources.Source do
retention_period_days retention_period_days
title_filter_regex title_filter_regex
media_profile_id media_profile_id
output_path_template_override
)a )a
# Expensive API calls are made when a source is inserted/updated so # Expensive API calls are made when a source is inserted/updated so
@@ -76,6 +77,7 @@ defmodule Pinchflat.Sources.Source do
field :retention_period_days, :integer field :retention_period_days, :integer
field :original_url, :string field :original_url, :string
field :title_filter_regex, :string field :title_filter_regex, :string
field :output_path_template_override, :string
field :series_directory, :string field :series_directory, :string
field :nfo_filepath, :string field :nfo_filepath, :string
@@ -109,6 +111,8 @@ defmodule Pinchflat.Sources.Source do
|> dynamic_default(:uuid, fn _ -> Ecto.UUID.generate() end) |> dynamic_default(:uuid, fn _ -> Ecto.UUID.generate() end)
|> validate_required(required_fields) |> validate_required(required_fields)
|> validate_number(:retention_period_days, greater_than_or_equal_to: 0) |> validate_number(:retention_period_days, greater_than_or_equal_to: 0)
# Ensures it ends with `.{{ ext }}` or `.%(ext)s` or similar (with a little wiggle room)
|> validate_format(:output_path_template_override, MediaProfile.ext_regex(), message: "must end with .{{ ext }}")
|> cast_assoc(:metadata, with: &SourceMetadata.changeset/2, required: false) |> cast_assoc(:metadata, with: &SourceMetadata.changeset/2, required: false)
|> unique_constraint([:collection_id, :media_profile_id, :title_filter_regex], error_key: :original_url) |> unique_constraint([:collection_id, :media_profile_id, :title_filter_regex], error_key: :original_url)
end end
+16 -3
View File
@@ -13,12 +13,25 @@ defmodule Pinchflat.Sources do
alias Pinchflat.Profiles.MediaProfile alias Pinchflat.Profiles.MediaProfile
alias Pinchflat.YtDlp.MediaCollection alias Pinchflat.YtDlp.MediaCollection
alias Pinchflat.Metadata.SourceMetadata alias Pinchflat.Metadata.SourceMetadata
alias Pinchflat.Filesystem.FilesystemHelpers alias Pinchflat.Utils.FilesystemUtils
alias Pinchflat.Downloading.DownloadingHelpers alias Pinchflat.Downloading.DownloadingHelpers
alias Pinchflat.FastIndexing.FastIndexingWorker alias Pinchflat.FastIndexing.FastIndexingWorker
alias Pinchflat.SlowIndexing.SlowIndexingHelpers alias Pinchflat.SlowIndexing.SlowIndexingHelpers
alias Pinchflat.Metadata.SourceMetadataStorageWorker alias Pinchflat.Metadata.SourceMetadataStorageWorker
@doc """
Returns the relevant output path template for a source.
Pulls from the source's override if present, otherwise uses the media profile's.
Returns binary()
"""
def output_path_template(source) do
source = Repo.preload(source, :media_profile)
media_profile = source.media_profile
source.output_path_template_override || media_profile.output_path_template
end
@doc """ @doc """
Returns the list of sources. Returns [%Source{}, ...] Returns the list of sources. Returns [%Source{}, ...]
""" """
@@ -153,7 +166,7 @@ defmodule Pinchflat.Sources do
Source.filepath_attributes() Source.filepath_attributes()
|> Enum.map(fn field -> mapped_struct[field] end) |> Enum.map(fn field -> mapped_struct[field] end)
|> Enum.filter(&is_binary/1) |> Enum.filter(&is_binary/1)
|> Enum.each(&FilesystemHelpers.delete_file_and_remove_empty_directories/1) |> Enum.each(&FilesystemUtils.delete_file_and_remove_empty_directories/1)
end end
defp delete_internal_metadata_files(source) do defp delete_internal_metadata_files(source) do
@@ -163,7 +176,7 @@ defmodule Pinchflat.Sources do
SourceMetadata.filepath_attributes() SourceMetadata.filepath_attributes()
|> Enum.map(fn field -> mapped_struct[field] end) |> Enum.map(fn field -> mapped_struct[field] end)
|> Enum.filter(&is_binary/1) |> Enum.filter(&is_binary/1)
|> Enum.each(&FilesystemHelpers.delete_file_and_remove_empty_directories/1) |> Enum.each(&FilesystemUtils.delete_file_and_remove_empty_directories/1)
end end
defp add_source_details_to_changeset(source, changeset) do defp add_source_details_to_changeset(source, changeset) do
+17
View File
@@ -5,6 +5,23 @@ defmodule Pinchflat.Utils.CliUtils do
alias Pinchflat.Utils.StringUtils alias Pinchflat.Utils.StringUtils
@doc """
Wraps a command in a shell script that will terminate
the command if stdin is closed. Useful for stopping
commands if the job runner is cancelled.
Delegates to `System.cmd/3` and any options/output
are passed through.
Returns {binary(), integer()}
"""
def wrap_cmd(command, args, opts \\ []) do
wrapper_command = Path.join(:code.priv_dir(:pinchflat), "cmd_wrapper.sh")
actual_command = [command] ++ args
System.cmd(wrapper_command, actual_command, opts)
end
@doc """ @doc """
Parses a list of command options into a list of strings suitable for passing to Parses a list of command options into a list of strings suitable for passing to
`System.cmd/3`. `System.cmd/3`.
@@ -1,4 +1,4 @@
defmodule Pinchflat.Filesystem.FilesystemHelpers do defmodule Pinchflat.Utils.FilesystemUtils do
@moduledoc """ @moduledoc """
Utility methods for working with the filesystem Utility methods for working with the filesystem
""" """
+21 -12
View File
@@ -7,7 +7,7 @@ defmodule Pinchflat.YtDlp.CommandRunner do
alias Pinchflat.Utils.CliUtils alias Pinchflat.Utils.CliUtils
alias Pinchflat.YtDlp.YtDlpCommandRunner alias Pinchflat.YtDlp.YtDlpCommandRunner
alias Pinchflat.Filesystem.FilesystemHelpers, as: FSUtils alias Pinchflat.Utils.FilesystemUtils, as: FSUtils
@behaviour YtDlpCommandRunner @behaviour YtDlpCommandRunner
@@ -27,16 +27,17 @@ defmodule Pinchflat.YtDlp.CommandRunner do
def run(url, command_opts, output_template, addl_opts \\ []) do def run(url, command_opts, output_template, addl_opts \\ []) do
# This approach lets us mock the command for testing # This approach lets us mock the command for testing
command = backend_executable() command = backend_executable()
# These must stay in exactly this order, hence why I'm giving it its own variable.
# Also, can't use RAM file since yt-dlp needs a concrete filepath.
output_filepath = generate_output_filepath(addl_opts) output_filepath = generate_output_filepath(addl_opts)
print_to_file_opts = [{:print_to_file, output_template}, output_filepath] print_to_file_opts = [{:print_to_file, output_template}, output_filepath]
cookie_opts = build_cookie_options() external_file_opts = build_external_file_options()
formatted_command_opts = [url] ++ CliUtils.parse_options(command_opts ++ print_to_file_opts ++ cookie_opts) # These must stay in exactly this order, hence why I'm giving it its own variable.
all_opts = command_opts ++ print_to_file_opts ++ external_file_opts
formatted_command_opts = [url] ++ CliUtils.parse_options(all_opts)
Logger.info("[yt-dlp] called with: #{Enum.join(formatted_command_opts, " ")}") Logger.info("[yt-dlp] called with: #{Enum.join(formatted_command_opts, " ")}")
case System.cmd(command, formatted_command_opts, stderr_to_stdout: true) do case CliUtils.wrap_cmd(command, formatted_command_opts, stderr_to_stdout: true) do
{_, 0} -> {_, 0} ->
# IDEA: consider deleting the file after reading it. It's in the tmp dir, so it's not # IDEA: consider deleting the file after reading it. It's in the tmp dir, so it's not
# a huge deal, but it's still a good idea to clean up after ourselves. # a huge deal, but it's still a good idea to clean up after ourselves.
@@ -73,17 +74,25 @@ defmodule Pinchflat.YtDlp.CommandRunner do
end end
end end
defp build_cookie_options do defp build_external_file_options do
base_dir = Application.get_env(:pinchflat, :extras_directory) base_dir = Application.get_env(:pinchflat, :extras_directory)
cookie_file = Path.join(base_dir, "cookies.txt") filename_options_map = %{cookies: "cookies.txt"}
case File.read(cookie_file) do Enum.reduce(filename_options_map, [], fn {opt_name, filename}, acc ->
{:ok, cookie_data} -> filepath = Path.join(base_dir, filename)
if String.trim(cookie_data) != "", do: [cookies: cookie_file], else: []
case File.read(filepath) do
{:ok, file_data} ->
if String.trim(file_data) != "" do
[{opt_name, filepath} | acc]
else
acc
end
{:error, _} -> {:error, _} ->
[] acc
end end
end)
end end
defp backend_executable do defp backend_executable do
+2 -2
View File
@@ -6,7 +6,7 @@ defmodule Pinchflat.YtDlp.MediaCollection do
require Logger require Logger
alias Pinchflat.Filesystem.FilesystemHelpers alias Pinchflat.Utils.FilesystemUtils
alias Pinchflat.YtDlp.Media, as: YtDlpMedia alias Pinchflat.YtDlp.Media, as: YtDlpMedia
@doc """ @doc """
@@ -26,7 +26,7 @@ defmodule Pinchflat.YtDlp.MediaCollection do
# available formats since we're just getting the media details # available formats since we're just getting the media details
command_opts = [:simulate, :skip_download, :ignore_no_formats_error] command_opts = [:simulate, :skip_download, :ignore_no_formats_error]
output_template = YtDlpMedia.indexing_output_template() output_template = YtDlpMedia.indexing_output_template()
output_filepath = FilesystemHelpers.generate_metadata_tmpfile(:json) output_filepath = FilesystemUtils.generate_metadata_tmpfile(:json)
file_listener_handler = Keyword.get(addl_opts, :file_listener_handler, false) file_listener_handler = Keyword.get(addl_opts, :file_listener_handler, false)
if file_listener_handler do if file_listener_handler do
@@ -203,6 +203,17 @@
/> />
</section> </section>
<section x-data="{ presets: { default: null, media_center: 1, audio: null, archiving: 1 } }">
<.input
field={f[:redownload_delay_days]}
type="number"
label="Redownload Delay (days)"
min="0"
help="Delay in days until new media is redownloaded. Redownloading new media can improve its quality or SponsorBlock tags. Leave blank to not redownload"
x-init="$watch('selectedPreset', p => p && ($el.value = presets[p]))"
/>
</section>
<h3 class="mt-8 text-2xl text-black dark:text-white"> <h3 class="mt-8 text-2xl text-black dark:text-white">
Media Center Options Media Center Options
</h3> </h3>
@@ -63,7 +63,7 @@ defmodule PinchflatWeb.Sources.SourceController do
pending_media = pending_media =
MediaQuery.new() MediaQuery.new()
|> MediaQuery.for_source(source) |> MediaQuery.for_source(source)
|> MediaQuery.with_media_pending_download() |> MediaQuery.where_pending_download()
|> order_by(desc: :id) |> order_by(desc: :id)
|> limit(100) |> limit(100)
|> Repo.all() |> Repo.all()
@@ -28,4 +28,20 @@ defmodule PinchflatWeb.Sources.SourceHTML do
def rss_feed_url(conn, source) do def rss_feed_url(conn, source) do
url(conn, ~p"/sources/#{source.uuid}/feed") <> ".xml" url(conn, ~p"/sources/#{source.uuid}/feed") <> ".xml"
end end
def output_path_template_override_placeholders(media_profiles) do
media_profiles
|> Enum.map(&{&1.id, &1.output_path_template})
|> Map.new()
|> Phoenix.json_library().encode!()
end
def output_path_template_override_help do
help_button_classes = "underline decoration-bodydark decoration-1 hover:decoration-white cursor-pointer"
help_button = ~s{<span class="#{help_button_classes}" x-on:click="$dispatch('load-template')">Click here</span>}
"""
Must end with .{{ ext }}. Same rules as Media Profile output path templates. #{help_button} to load your media profile's output template
"""
end
end end
@@ -9,6 +9,7 @@
Oops, something went wrong! Please check the errors below. Oops, something went wrong! Please check the errors below.
</.error> </.error>
<section x-data="{ mediaProfileId: null }">
<section class="flex justify-between items-center mt-8"> <section class="flex justify-between items-center mt-8">
<h3 class=" text-2xl text-black dark:text-white"> <h3 class=" text-2xl text-black dark:text-white">
General Options General Options
@@ -33,6 +34,7 @@
type="select" type="select"
label="Media Profile" label="Media Profile"
help="Sets your preferences for what media to look for and how to store it" help="Sets your preferences for what media to look for and how to store it"
x-model.fill="mediaProfileId"
/> />
<h3 class="mt-8 text-2xl text-black dark:text-white"> <h3 class="mt-8 text-2xl text-black dark:text-white">
@@ -111,6 +113,29 @@
placeholder="(?i)^How to Bike$" placeholder="(?i)^How to Bike$"
help="A PCRE-compatible regex. Only media with titles that match this regex will be downloaded. Look up 'SQLean Regex docs' for more" help="A PCRE-compatible regex. Only media with titles that match this regex will be downloaded. Look up 'SQLean Regex docs' for more"
/> />
<section
x-data={
"""
{
placeholders: JSON.parse('#{output_path_template_override_placeholders(@media_profiles)}'),
inputValue: null
}
"""
}
x-on:load-template="inputValue = placeholders[mediaProfileId]"
>
<.input
field={f[:output_path_template_override]}
type="text"
inputclass="font-mono"
label="Output path template override"
help={output_path_template_override_help()}
html_help={true}
x-bind:placeholder="placeholders[mediaProfileId]"
x-model.fill="inputValue"
/>
</section>
</section> </section>
<.button class="my-10 sm:mb-7.5 w-full sm:w-auto" rounding="rounded-lg">Save Source</.button> <.button class="my-10 sm:mb-7.5 w-full sm:w-auto" rounding="rounded-lg">Save Source</.button>
@@ -118,4 +143,5 @@
<div class="rounded-sm dark:bg-meta-4 p-4 md:p-6 mb-5"> <div class="rounded-sm dark:bg-meta-4 p-4 md:p-6 mb-5">
<.fast_indexing_help /> <.fast_indexing_help />
</div> </div>
</section>
</.simple_form> </.simple_form>
+1 -1
View File
@@ -4,7 +4,7 @@ defmodule Pinchflat.MixProject do
def project do def project do
[ [
app: :pinchflat, app: :pinchflat,
version: "0.1.10", version: "0.1.13",
elixir: "~> 1.16", elixir: "~> 1.16",
elixirc_paths: elixirc_paths(Mix.env()), elixirc_paths: elixirc_paths(Mix.env()),
start_permanent: Mix.env() == :prod, start_permanent: Mix.env() == :prod,
+26
View File
@@ -0,0 +1,26 @@
#!/usr/bin/env bash
# This script is a wrapper for other programs
# that ensures they are killed when stdin closes
# (eg: a job terminates)
# Start the program in the background
exec "$@" &
pid1=$!
# Silence warnings from here on
exec >/dev/null 2>&1
# Read from stdin in the background and
# kill running program when stdin closes
exec 0<&0 $(
while read; do :; done
kill -KILL $pid1
) &
pid2=$!
# Clean up
wait $pid1
ret=$?
kill -KILL $pid2
exit $ret
@@ -0,0 +1,13 @@
defmodule Pinchflat.Repo.Migrations.AddRedownloadedFields do
use Ecto.Migration
def change do
alter table(:media_profiles) do
add :redownload_delay_days, :integer
end
alter table(:media_items) do
add :media_redownloaded_at, :utc_datetime
end
end
end
@@ -0,0 +1,9 @@
defmodule Pinchflat.Repo.Migrations.AddOutputTemplateToSources do
use Ecto.Migration
def change do
alter table(:sources) do
add :output_path_template_override, :string
end
end
end
@@ -27,7 +27,7 @@ defmodule Pinchflat.Boot.PreJobStartupTasksTest do
end end
end end
describe "create_blank_cookie_file" do describe "create_blank_yt_dlp_files" do
test "creates a blank cookie file" do test "creates a blank cookie file" do
base_dir = Application.get_env(:pinchflat, :extras_directory) base_dir = Application.get_env(:pinchflat, :extras_directory)
filepath = Path.join(base_dir, "cookies.txt") filepath = Path.join(base_dir, "cookies.txt")
@@ -39,6 +39,18 @@ defmodule Pinchflat.Boot.PreJobStartupTasksTest do
assert File.exists?(filepath) assert File.exists?(filepath)
end end
test "creates a blank yt-dlp config file" do
base_dir = Application.get_env(:pinchflat, :extras_directory)
filepath = Path.join([base_dir, "yt-dlp-configs", "base-config.txt"])
File.rm(filepath)
refute File.exists?(filepath)
PreJobStartupTasks.init(%{})
assert File.exists?(filepath)
end
end end
describe "apply_default_settings" do describe "apply_default_settings" do
@@ -4,7 +4,9 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilderTest do
import Pinchflat.SourcesFixtures import Pinchflat.SourcesFixtures
import Pinchflat.ProfilesFixtures import Pinchflat.ProfilesFixtures
alias Pinchflat.Sources
alias Pinchflat.Profiles alias Pinchflat.Profiles
alias Pinchflat.Utils.FilesystemUtils
alias Pinchflat.Downloading.DownloadOptionBuilder alias Pinchflat.Downloading.DownloadOptionBuilder
setup do setup do
@@ -30,6 +32,20 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilderTest do
assert {:output, "/tmp/test/media/#{media_item.source.custom_name}.%(ext)s"} in res assert {:output, "/tmp/test/media/#{media_item.source.custom_name}.%(ext)s"} in res
end end
test "uses source's output override if present", %{media_item: media_item} do
source = media_item.source
{:ok, _} = Sources.update_source(source, %{output_path_template_override: "override.%(ext)s"})
media_item =
media_item
|> Repo.reload()
|> Repo.preload(source: :media_profile)
assert {:ok, res} = DownloadOptionBuilder.build(media_item)
assert {:output, "/tmp/test/media/override.%(ext)s"} in res
end
end end
describe "build/1 when testing default options" do describe "build/1 when testing default options" do
@@ -38,6 +54,7 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilderTest do
assert :no_progress in res assert :no_progress in res
assert :windows_filenames in res assert :windows_filenames in res
assert :force_overwrites in res
end end
end end
@@ -133,6 +150,20 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilderTest do
assert {:output, "thumbnail:/tmp/test/media/%(title)S-thumb.%(ext)s"} in res assert {:output, "thumbnail:/tmp/test/media/%(title)S-thumb.%(ext)s"} in res
end end
test "appends -thumb to source's output path override, if present", %{media_item: media_item} do
media_item = update_media_profile_attribute(media_item, %{download_thumbnail: true})
{:ok, _} = Sources.update_source(media_item.source, %{output_path_template_override: "override.%(ext)s"})
media_item =
media_item
|> Repo.reload()
|> Repo.preload(source: :media_profile)
assert {:ok, res} = DownloadOptionBuilder.build(media_item)
assert {:output, "thumbnail:/tmp/test/media/override-thumb.%(ext)s"} in res
end
test "converts thumbnail to jpg when download_thumbnail is true", %{media_item: media_item} do test "converts thumbnail to jpg when download_thumbnail is true", %{media_item: media_item} do
media_item = update_media_profile_attribute(media_item, %{download_thumbnail: true}) media_item = update_media_profile_attribute(media_item, %{download_thumbnail: true})
@@ -209,6 +240,7 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilderTest do
assert {:ok, res} = DownloadOptionBuilder.build(media_item) assert {:ok, res} = DownloadOptionBuilder.build(media_item)
assert {:format_sort, "res:#{resolution},+codec:avc:m4a"} in res assert {:format_sort, "res:#{resolution},+codec:avc:m4a"} in res
assert {:remux_video, "mp4"} in res
end) end)
end end
@@ -219,6 +251,8 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilderTest do
assert :extract_audio in res assert :extract_audio in res
assert {:format, "bestaudio[ext=m4a]"} in res assert {:format, "bestaudio[ext=m4a]"} in res
refute {:remux_video, "mp4"} in res
end end
end end
@@ -261,12 +295,111 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilderTest do
end end
end end
describe "build/1 when testing config file options" do
setup do
base_dir = Path.join(Application.get_env(:pinchflat, :extras_directory), "yt-dlp-configs")
{:ok, %{base_dir: base_dir}}
end
test "includes base config file if it's present", %{media_item: media_item, base_dir: base_dir} do
filepath = Path.join(base_dir, "base-config.txt")
FilesystemUtils.write_p!(filepath, "base config")
assert {:ok, res} = DownloadOptionBuilder.build(media_item)
assert {:config_locations, filepath} in res
end
test "includes media profile config file if it's present", %{media_item: media_item, base_dir: base_dir} do
media_profile = media_item.source.media_profile
filepath = Path.join(base_dir, "media-profile-#{media_profile.id}-config.txt")
FilesystemUtils.write_p!(filepath, "profile config")
assert {:ok, res} = DownloadOptionBuilder.build(media_item)
assert {:config_locations, filepath} in res
end
test "includes source config file if it's present", %{media_item: media_item, base_dir: base_dir} do
source = media_item.source
filepath = Path.join(base_dir, "source-#{source.id}-config.txt")
FilesystemUtils.write_p!(filepath, "profile config")
assert {:ok, res} = DownloadOptionBuilder.build(media_item)
assert {:config_locations, filepath} in res
end
test "includes media item config file if it's present", %{media_item: media_item, base_dir: base_dir} do
filepath = Path.join(base_dir, "media-item-#{media_item.id}-config.txt")
FilesystemUtils.write_p!(filepath, "media item config")
assert {:ok, res} = DownloadOptionBuilder.build(media_item)
assert {:config_locations, filepath} in res
end
test "does not include config file options if they are not present", %{media_item: media_item} do
assert {:ok, res} = DownloadOptionBuilder.build(media_item)
refute :config_locations in res
end
test "does not return a config file if it's blank", %{media_item: media_item, base_dir: base_dir} do
filepath = Path.join(base_dir, "base-config.txt")
FilesystemUtils.write_p!(filepath, " \n \n ")
assert {:ok, res} = DownloadOptionBuilder.build(media_item)
refute :config_locations in res
end
test "returns config files in order of precedence", %{media_item: media_item, base_dir: base_dir} do
source = media_item.source
media_profile = source.media_profile
base_filepath = Path.join(base_dir, "base-config.txt")
source_filepath = Path.join(base_dir, "source-#{source.id}-config.txt")
media_item_filepath = Path.join(base_dir, "media-item-#{media_item.id}-config.txt")
media_profile_filepath = Path.join(base_dir, "media-profile-#{media_profile.id}-config.txt")
FilesystemUtils.write_p!(base_filepath, "config")
FilesystemUtils.write_p!(source_filepath, "config")
FilesystemUtils.write_p!(media_item_filepath, "config")
FilesystemUtils.write_p!(media_profile_filepath, "config")
assert {:ok, res} = DownloadOptionBuilder.build(media_item)
expected_order = [
{:config_locations, base_filepath},
{:config_locations, media_profile_filepath},
{:config_locations, source_filepath},
{:config_locations, media_item_filepath}
]
assert Enum.filter(res, fn
{:config_locations, _} -> true
_ -> false
end) == expected_order
end
end
describe "build_output_path_for/1" do describe "build_output_path_for/1" do
test "builds an output path for a source", %{media_item: media_item} do test "builds an output path for a source", %{media_item: media_item} do
path = DownloadOptionBuilder.build_output_path_for(media_item.source) path = DownloadOptionBuilder.build_output_path_for(media_item.source)
assert path == "/tmp/test/media/%(title)S.%(ext)s" assert path == "/tmp/test/media/%(title)S.%(ext)s"
end end
test "uses source's output override if present", %{media_item: media_item} do
source = media_item.source
{:ok, source} = Sources.update_source(source, %{output_path_template_override: "override.%(ext)s"})
path = DownloadOptionBuilder.build_output_path_for(source)
assert path == "/tmp/test/media/override.%(ext)s"
end
end end
defp update_media_profile_attribute(media_item_with_preloads, attrs) do defp update_media_profile_attribute(media_item_with_preloads, attrs) do
@@ -6,7 +6,7 @@ defmodule Pinchflat.Downloading.MediaDownloadWorkerTest do
alias Pinchflat.Media alias Pinchflat.Media
alias Pinchflat.Sources alias Pinchflat.Sources
alias Pinchflat.Filesystem.FilesystemHelpers alias Pinchflat.Utils.FilesystemUtils
alias Pinchflat.Downloading.MediaDownloadWorker alias Pinchflat.Downloading.MediaDownloadWorker
setup :verify_on_exit! setup :verify_on_exit!
@@ -62,7 +62,9 @@ defmodule Pinchflat.Downloading.MediaDownloadWorkerTest do
assert media_item.media_filepath == nil assert media_item.media_filepath == nil
perform_job(MediaDownloadWorker, %{id: media_item.id}) perform_job(MediaDownloadWorker, %{id: media_item.id})
assert Repo.reload(media_item).media_filepath != nil media_item = Repo.reload(media_item)
assert media_item.media_filepath != nil
end end
test "it saves the metadata to the media_item", %{media_item: media_item} do test "it saves the metadata to the media_item", %{media_item: media_item} do
@@ -138,7 +140,7 @@ defmodule Pinchflat.Downloading.MediaDownloadWorkerTest do
test "it saves the file's size to the database", %{media_item: media_item} do test "it saves the file's size to the database", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl -> expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl ->
metadata = render_parsed_metadata(:media_metadata) metadata = render_parsed_metadata(:media_metadata)
FilesystemHelpers.write_p!(metadata["filepath"], "test") FilesystemUtils.write_p!(metadata["filepath"], "test")
{:ok, Phoenix.json_library().encode!(metadata)} {:ok, Phoenix.json_library().encode!(metadata)}
end) end)
@@ -149,6 +151,28 @@ defmodule Pinchflat.Downloading.MediaDownloadWorkerTest do
assert media_item.media_size_bytes > 0 assert media_item.media_size_bytes > 0
end end
test "saves redownloaded_at if this is for a redownload", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl ->
{:ok, render_metadata(:media_metadata)}
end)
perform_job(MediaDownloadWorker, %{id: media_item.id, redownload?: true})
media_item = Repo.reload(media_item)
assert media_item.media_redownloaded_at != nil
end
test "doesn't save redownloaded_at if this is not for a redownload", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl ->
{:ok, render_metadata(:media_metadata)}
end)
perform_job(MediaDownloadWorker, %{id: media_item.id})
media_item = Repo.reload(media_item)
assert media_item.media_redownloaded_at == nil
end
test "does not blow up if the record doesn't exist" do test "does not blow up if the record doesn't exist" do
assert :ok = perform_job(MediaDownloadWorker, %{id: 0}) assert :ok = perform_job(MediaDownloadWorker, %{id: 0})
end end
@@ -0,0 +1,44 @@
defmodule Pinchflat.Downloading.MediaRedownloadWorkerTest do
use Pinchflat.DataCase
import Pinchflat.MediaFixtures
import Pinchflat.SourcesFixtures
import Pinchflat.ProfilesFixtures
alias Pinchflat.Downloading.MediaDownloadWorker
alias Pinchflat.Downloading.MediaRedownloadWorker
describe "perform/1" do
test "kicks off a task for redownloadable media items" do
media_profile = media_profile_fixture(%{redownload_delay_days: 4})
source = source_fixture(%{media_profile_id: media_profile.id, inserted_at: now_minus(10, :days)})
media_item =
media_item_fixture(%{
source_id: source.id,
upload_date: now_minus(6, :days),
media_downloaded_at: now_minus(5, :days)
})
perform_job(MediaRedownloadWorker, %{})
assert [_] = all_enqueued(worker: MediaDownloadWorker, args: %{id: media_item.id, redownload?: true})
end
test "does not kickoff a task for non-redownloadable media items" do
media_profile = media_profile_fixture(%{redownload_delay_days: 4})
source = source_fixture(%{media_profile_id: media_profile.id, inserted_at: now_minus(10, :days)})
_media_item =
media_item_fixture(%{
source_id: source.id,
upload_date: now_minus(6, :days),
media_downloaded_at: now_minus(1, :day)
})
perform_job(MediaRedownloadWorker, %{})
assert [] = all_enqueued(worker: MediaDownloadWorker)
end
end
end
+125
View File
@@ -130,6 +130,131 @@ defmodule Pinchflat.MediaTest do
end end
end end
describe "list_redownloadable_media_items/0" do
setup do
media_profile = media_profile_fixture(%{redownload_delay_days: 4})
source = source_fixture(%{media_profile_id: media_profile.id, inserted_at: now_minus(10, :days)})
{:ok, %{media_profile: media_profile, source: source}}
end
test "returns media eligible for redownload", %{source: source} do
media_item =
media_item_fixture(%{
source_id: source.id,
upload_date: now_minus(6, :days),
media_downloaded_at: now_minus(5, :days)
})
assert Media.list_redownloadable_media_items() == [media_item]
end
test "returns media items that were downloaded in past but still meet redownload delay", %{source: source} do
media_item =
media_item_fixture(%{
source_id: source.id,
upload_date: now_minus(20, :days),
media_downloaded_at: now_minus(19, :days)
})
assert Media.list_redownloadable_media_items() == [media_item]
end
test "does not return media items without a media_downloaded_at", %{source: source} do
_media_item =
media_item_fixture(%{
source_id: source.id,
upload_date: now_minus(5, :days),
media_downloaded_at: nil
})
assert Media.list_redownloadable_media_items() == []
end
test "does not return media items that are set to prevent download", %{source: source} do
_media_item =
media_item_fixture(%{
source_id: source.id,
upload_date: now_minus(5, :days),
media_downloaded_at: now(),
prevent_download: true
})
assert Media.list_redownloadable_media_items() == []
end
test "does not return media items that have been culled", %{source: source} do
_media_item =
media_item_fixture(%{
source_id: source.id,
upload_date: now_minus(5, :days),
media_downloaded_at: now(),
culled_at: now()
})
assert Media.list_redownloadable_media_items() == []
end
test "does not return media items before the download delay", %{source: source} do
_media_item =
media_item_fixture(%{
source_id: source.id,
upload_date: now_minus(3, :days),
media_downloaded_at: now_minus(3, :days)
})
assert Media.list_redownloadable_media_items() == []
end
test "does not return media items that have already been redownloaded", %{source: source} do
_media_item =
media_item_fixture(%{
source_id: source.id,
upload_date: now_minus(5, :days),
media_downloaded_at: now(),
media_redownloaded_at: now()
})
assert Media.list_redownloadable_media_items() == []
end
test "does not return media items that were first downloaded well after the upload_date", %{source: source} do
_media_item =
media_item_fixture(%{
source_id: source.id,
media_downloaded_at: now(),
upload_date: now_minus(20, :days)
})
assert Media.list_redownloadable_media_items() == []
end
test "does not return media items that were recently uploaded", %{source: source} do
_media_item =
media_item_fixture(%{
source_id: source.id,
media_downloaded_at: now(),
upload_date: now_minus(2, :days)
})
assert Media.list_redownloadable_media_items() == []
end
test "does not return media items without a redownload delay" do
media_profile = media_profile_fixture(%{redownload_delay_days: nil})
source = source_fixture(%{media_profile_id: media_profile.id})
_media_item =
media_item_fixture(%{
source_id: source.id,
upload_date: now_minus(6, :days),
media_downloaded_at: now_minus(5, :days)
})
assert Media.list_redownloadable_media_items() == []
end
end
describe "list_pending_media_items_for/1" do describe "list_pending_media_items_for/1" do
test "it returns pending without a filepath for a given source" do test "it returns pending without a filepath for a given source" do
source = source_fixture() source = source_fixture()
+2 -2
View File
@@ -2,10 +2,10 @@ defmodule Pinchflat.Metadata.NfoBuilderTest do
use Pinchflat.DataCase use Pinchflat.DataCase
alias Pinchflat.Metadata.NfoBuilder alias Pinchflat.Metadata.NfoBuilder
alias Pinchflat.Filesystem.FilesystemHelpers alias Pinchflat.Utils.FilesystemUtils
setup do setup do
filepath = FilesystemHelpers.generate_metadata_tmpfile(:json) filepath = FilesystemUtils.generate_metadata_tmpfile(:json)
on_exit(fn -> File.rm!(filepath) end) on_exit(fn -> File.rm!(filepath) end)
@@ -1,12 +1,12 @@
defmodule Pinchflat.SlowIndexing.FileFollowerServerTest do defmodule Pinchflat.SlowIndexing.FileFollowerServerTest do
use ExUnit.Case, async: true use ExUnit.Case, async: true
alias alias Pinchflat.Filesystem.FilesystemHelpers alias alias Pinchflat.Utils.FilesystemUtils
alias Pinchflat.SlowIndexing.FileFollowerServer alias Pinchflat.SlowIndexing.FileFollowerServer
setup do setup do
{:ok, pid} = FileFollowerServer.start_link() {:ok, pid} = FileFollowerServer.start_link()
tmpfile = FilesystemHelpers.generate_metadata_tmpfile(:txt) tmpfile = FilesystemUtils.generate_metadata_tmpfile(:txt)
{:ok, %{pid: pid, tmpfile: tmpfile}} {:ok, %{pid: pid, tmpfile: tmpfile}}
end end
+25 -3
View File
@@ -8,7 +8,7 @@ defmodule Pinchflat.SourcesTest do
alias Pinchflat.Sources alias Pinchflat.Sources
alias Pinchflat.Sources.Source alias Pinchflat.Sources.Source
alias Pinchflat.Filesystem.FilesystemHelpers alias Pinchflat.Utils.FilesystemUtils
alias Pinchflat.Metadata.MetadataFileHelpers alias Pinchflat.Metadata.MetadataFileHelpers
alias Pinchflat.Downloading.DownloadingHelpers alias Pinchflat.Downloading.DownloadingHelpers
alias Pinchflat.FastIndexing.FastIndexingWorker alias Pinchflat.FastIndexing.FastIndexingWorker
@@ -35,6 +35,28 @@ defmodule Pinchflat.SourcesTest do
end end
end end
describe "output_path_template/1" do
test "returns the source's override if present" do
source = source_fixture(%{output_path_template_override: "/override/{{ title }}.{{ ext }}"})
assert Sources.output_path_template(source) == "/override/{{ title }}.{{ ext }}"
end
test "returns the media profile's template if no override is present" do
media_profile = media_profile_fixture(%{output_path_template: "/profile/{{ title }}.{{ ext }}"})
source = source_fixture(%{media_profile_id: media_profile.id})
assert Sources.output_path_template(source) == "/profile/{{ title }}.{{ ext }}"
end
test "Treats empty strings as being blank" do
media_profile = media_profile_fixture(%{output_path_template: "/profile/{{ title }}.{{ ext }}"})
source = source_fixture(%{media_profile_id: media_profile.id, output_path_template_override: " "})
assert Sources.output_path_template(source) == "/profile/{{ title }}.{{ ext }}"
end
end
describe "list_sources/0" do describe "list_sources/0" do
test "it returns all sources" do test "it returns all sources" do
source = source_fixture() source = source_fixture()
@@ -561,7 +583,7 @@ defmodule Pinchflat.SourcesTest do
end end
test "does not delete the source's non-metadata files" do test "does not delete the source's non-metadata files" do
filepath = FilesystemHelpers.generate_metadata_tmpfile(:nfo) filepath = FilesystemUtils.generate_metadata_tmpfile(:nfo)
source = source_fixture(%{nfo_filepath: filepath}) source = source_fixture(%{nfo_filepath: filepath})
assert {:ok, _} = Sources.delete_source(source) assert {:ok, _} = Sources.delete_source(source)
@@ -592,7 +614,7 @@ defmodule Pinchflat.SourcesTest do
end end
test "deletes the source's non-metadata files" do test "deletes the source's non-metadata files" do
filepath = FilesystemHelpers.generate_metadata_tmpfile(:nfo) filepath = FilesystemUtils.generate_metadata_tmpfile(:nfo)
source = source_fixture(%{nfo_filepath: filepath}) source = source_fixture(%{nfo_filepath: filepath})
assert {:ok, _} = Sources.delete_source(source, delete_files: true) assert {:ok, _} = Sources.delete_source(source, delete_files: true)
+6
View File
@@ -3,6 +3,12 @@ defmodule Pinchflat.Utils.CliUtilsTest do
alias Pinchflat.Utils.CliUtils alias Pinchflat.Utils.CliUtils
describe "wrap_cmd/3" do
test "delegates to System.cmd/3" do
assert {"output\n", 0} = CliUtils.wrap_cmd("echo", ["output"])
end
end
describe "parse_options/1" do describe "parse_options/1" do
test "it converts symbol k-v arg keys to kebab case" do test "it converts symbol k-v arg keys to kebab case" do
assert ["--buffer-size", "1024"] = CliUtils.parse_options(buffer_size: 1024) assert ["--buffer-size", "1024"] = CliUtils.parse_options(buffer_size: 1024)
@@ -1,13 +1,13 @@
defmodule Pinchflat.Filesystem.FilesystemHelpersTest do defmodule Pinchflat.Utils.FilesystemUtilsTest do
use Pinchflat.DataCase use Pinchflat.DataCase
import Pinchflat.MediaFixtures import Pinchflat.MediaFixtures
alias Pinchflat.Filesystem.FilesystemHelpers alias Pinchflat.Utils.FilesystemUtils
describe "generate_metadata_tmpfile/1" do describe "generate_metadata_tmpfile/1" do
test "creates a tmpfile and returns its path" do test "creates a tmpfile and returns its path" do
res = FilesystemHelpers.generate_metadata_tmpfile(:json) res = FilesystemUtils.generate_metadata_tmpfile(:json)
assert String.ends_with?(res, ".json") assert String.ends_with?(res, ".json")
assert File.exists?(res) assert File.exists?(res)
@@ -22,7 +22,7 @@ defmodule Pinchflat.Filesystem.FilesystemHelpersTest do
refute media_item.media_size_bytes refute media_item.media_size_bytes
assert {:ok, media_item} = FilesystemHelpers.compute_and_save_media_filesize(media_item) assert {:ok, media_item} = FilesystemUtils.compute_and_save_media_filesize(media_item)
assert Repo.reload!(media_item).media_size_bytes assert Repo.reload!(media_item).media_size_bytes
end end
@@ -30,16 +30,16 @@ defmodule Pinchflat.Filesystem.FilesystemHelpersTest do
test "returns the error if operation fails" do test "returns the error if operation fails" do
media_item = media_item_fixture(%{media_filepath: "/nonexistent/file.mkv"}) media_item = media_item_fixture(%{media_filepath: "/nonexistent/file.mkv"})
assert {:error, _} = FilesystemHelpers.compute_and_save_media_filesize(media_item) assert {:error, _} = FilesystemUtils.compute_and_save_media_filesize(media_item)
end end
end end
describe "write_p/3" do describe "write_p/3" do
test "writes content to a file" do test "writes content to a file" do
filepath = FilesystemHelpers.generate_metadata_tmpfile(:json) filepath = FilesystemUtils.generate_metadata_tmpfile(:json)
content = "{}" content = "{}"
assert :ok = FilesystemHelpers.write_p(filepath, content) assert :ok = FilesystemUtils.write_p(filepath, content)
assert File.read!(filepath) == content assert File.read!(filepath) == content
File.rm!(filepath) File.rm!(filepath)
@@ -50,7 +50,7 @@ defmodule Pinchflat.Filesystem.FilesystemHelpersTest do
filepath = Path.join([tmpfile_directory, "foo", "bar", "file.json"]) filepath = Path.join([tmpfile_directory, "foo", "bar", "file.json"])
content = "{}" content = "{}"
assert :ok = FilesystemHelpers.write_p(filepath, content) assert :ok = FilesystemUtils.write_p(filepath, content)
assert File.read!(filepath) == content assert File.read!(filepath) == content
File.rm!(filepath) File.rm!(filepath)
@@ -59,10 +59,10 @@ defmodule Pinchflat.Filesystem.FilesystemHelpersTest do
describe "write_p!/3" do describe "write_p!/3" do
test "writes content to a file" do test "writes content to a file" do
filepath = FilesystemHelpers.generate_metadata_tmpfile(:json) filepath = FilesystemUtils.generate_metadata_tmpfile(:json)
content = "{}" content = "{}"
assert :ok = FilesystemHelpers.write_p!(filepath, content) assert :ok = FilesystemUtils.write_p!(filepath, content)
assert File.read!(filepath) == content assert File.read!(filepath) == content
File.rm!(filepath) File.rm!(filepath)
@@ -73,7 +73,7 @@ defmodule Pinchflat.Filesystem.FilesystemHelpersTest do
filepath = Path.join([tmpfile_directory, "foo", "bar", "file.json"]) filepath = Path.join([tmpfile_directory, "foo", "bar", "file.json"])
content = "{}" content = "{}"
assert :ok = FilesystemHelpers.write_p!(filepath, content) assert :ok = FilesystemUtils.write_p!(filepath, content)
assert File.read!(filepath) == content assert File.read!(filepath) == content
File.rm!(filepath) File.rm!(filepath)
@@ -82,11 +82,11 @@ defmodule Pinchflat.Filesystem.FilesystemHelpersTest do
describe "delete_file_and_remove_empty_directories/1" do describe "delete_file_and_remove_empty_directories/1" do
test "deletes file at the provided filepath" do test "deletes file at the provided filepath" do
filepath = FilesystemHelpers.generate_metadata_tmpfile(:json) filepath = FilesystemUtils.generate_metadata_tmpfile(:json)
assert File.exists?(filepath) assert File.exists?(filepath)
assert :ok = FilesystemHelpers.delete_file_and_remove_empty_directories(filepath) assert :ok = FilesystemUtils.delete_file_and_remove_empty_directories(filepath)
refute File.exists?(filepath) refute File.exists?(filepath)
end end
@@ -94,9 +94,9 @@ defmodule Pinchflat.Filesystem.FilesystemHelpersTest do
test "deletes empty directories" do test "deletes empty directories" do
tmpfile_directory = Application.get_env(:pinchflat, :tmpfile_directory) tmpfile_directory = Application.get_env(:pinchflat, :tmpfile_directory)
filepath = Path.join([tmpfile_directory, "foo", "bar", "baz", "qux.json"]) filepath = Path.join([tmpfile_directory, "foo", "bar", "baz", "qux.json"])
FilesystemHelpers.write_p!(filepath, "") FilesystemUtils.write_p!(filepath, "")
assert :ok = FilesystemHelpers.delete_file_and_remove_empty_directories(filepath) assert :ok = FilesystemUtils.delete_file_and_remove_empty_directories(filepath)
refute File.exists?(filepath) refute File.exists?(filepath)
refute File.exists?(Path.join([tmpfile_directory, "foo", "bar", "baz"])) refute File.exists?(Path.join([tmpfile_directory, "foo", "bar", "baz"]))
@@ -108,10 +108,10 @@ defmodule Pinchflat.Filesystem.FilesystemHelpersTest do
tmpfile_directory = Application.get_env(:pinchflat, :tmpfile_directory) tmpfile_directory = Application.get_env(:pinchflat, :tmpfile_directory)
filepath_1 = Path.join([tmpfile_directory, "foo", "bar", "baz", "qux.json"]) filepath_1 = Path.join([tmpfile_directory, "foo", "bar", "baz", "qux.json"])
filepath_2 = Path.join([tmpfile_directory, "foo", "baz.json"]) filepath_2 = Path.join([tmpfile_directory, "foo", "baz.json"])
FilesystemHelpers.write_p!(filepath_1, "") FilesystemUtils.write_p!(filepath_1, "")
FilesystemHelpers.write_p!(filepath_2, "") FilesystemUtils.write_p!(filepath_2, "")
assert :ok = FilesystemHelpers.delete_file_and_remove_empty_directories(filepath_1) assert :ok = FilesystemUtils.delete_file_and_remove_empty_directories(filepath_1)
refute File.exists?(filepath_1) refute File.exists?(filepath_1)
refute File.exists?(Path.join([tmpfile_directory, "foo", "bar", "baz"])) refute File.exists?(Path.join([tmpfile_directory, "foo", "bar", "baz"]))
@@ -121,24 +121,24 @@ defmodule Pinchflat.Filesystem.FilesystemHelpersTest do
assert File.exists?(Path.join([tmpfile_directory, "foo"])) assert File.exists?(Path.join([tmpfile_directory, "foo"]))
# cleanup # cleanup
FilesystemHelpers.delete_file_and_remove_empty_directories(filepath_2) FilesystemUtils.delete_file_and_remove_empty_directories(filepath_2)
end end
test "returns an error if file could not be deleted" do test "returns an error if file could not be deleted" do
filepath = "/nonexistent/file.json" filepath = "/nonexistent/file.json"
assert {:error, _} = FilesystemHelpers.delete_file_and_remove_empty_directories(filepath) assert {:error, _} = FilesystemUtils.delete_file_and_remove_empty_directories(filepath)
end end
end end
describe "cp_p!/2" do describe "cp_p!/2" do
test "copies a file from source to destination" do test "copies a file from source to destination" do
source = "#{tmpfile_directory()}/source.json" source = "#{tmpfile_directory()}/source.json"
FilesystemHelpers.write_p!(source, "TEST") FilesystemUtils.write_p!(source, "TEST")
destination = "#{tmpfile_directory()}/destination.json" destination = "#{tmpfile_directory()}/destination.json"
refute File.exists?(destination) refute File.exists?(destination)
FilesystemHelpers.cp_p!(source, destination) FilesystemUtils.cp_p!(source, destination)
assert File.exists?(destination) assert File.exists?(destination)
assert File.read!(destination) == "TEST" assert File.read!(destination) == "TEST"
@@ -148,11 +148,11 @@ defmodule Pinchflat.Filesystem.FilesystemHelpersTest do
test "creates directories as needed" do test "creates directories as needed" do
source = "#{tmpfile_directory()}/source.json" source = "#{tmpfile_directory()}/source.json"
FilesystemHelpers.write_p!(source, "TEST") FilesystemUtils.write_p!(source, "TEST")
destination = "#{tmpfile_directory()}/foo/bar/destination.json" destination = "#{tmpfile_directory()}/foo/bar/destination.json"
refute File.exists?(destination) refute File.exists?(destination)
FilesystemHelpers.cp_p!(source, destination) FilesystemUtils.cp_p!(source, destination)
assert File.exists?(destination) assert File.exists?(destination)
File.rm!(source) File.rm!(source)
@@ -1,7 +1,7 @@
defmodule Pinchflat.YtDlp.CommandRunnerTest do defmodule Pinchflat.YtDlp.CommandRunnerTest do
use ExUnit.Case, async: true use ExUnit.Case, async: true
alias Pinchflat.Filesystem.FilesystemHelpers alias Pinchflat.Utils.FilesystemUtils
alias Pinchflat.YtDlp.CommandRunner, as: Runner alias Pinchflat.YtDlp.CommandRunner, as: Runner
@@ -42,16 +42,17 @@ defmodule Pinchflat.YtDlp.CommandRunnerTest do
end end
end end
describe "run/4 when testing cookie options" do describe "run/4 when testing external file options" do
setup do setup do
base_dir = Application.get_env(:pinchflat, :extras_directory) base_dir = Application.get_env(:pinchflat, :extras_directory)
cookie_file = Path.join(base_dir, "cookies.txt") cookie_file = Path.join(base_dir, "cookies.txt")
yt_dlp_file = Path.join([base_dir, "yt-dlp-configs", "main.txt"])
{:ok, cookie_file: cookie_file} {:ok, cookie_file: cookie_file, yt_dlp_file: yt_dlp_file}
end end
test "includes cookie options when cookies.txt exists", %{cookie_file: cookie_file} do test "includes cookie options when cookies.txt exists", %{cookie_file: cookie_file} do
FilesystemHelpers.write_p!(cookie_file, "cookie data") FilesystemUtils.write_p!(cookie_file, "cookie data")
assert {:ok, output} = Runner.run(@media_url, [], "") assert {:ok, output} = Runner.run(@media_url, [], "")
@@ -59,7 +60,7 @@ defmodule Pinchflat.YtDlp.CommandRunnerTest do
end end
test "doesn't include cookie options when cookies.txt blank", %{cookie_file: cookie_file} do test "doesn't include cookie options when cookies.txt blank", %{cookie_file: cookie_file} do
FilesystemHelpers.write_p!(cookie_file, " \n \n ") FilesystemUtils.write_p!(cookie_file, " \n \n ")
assert {:ok, output} = Runner.run(@media_url, [], "") assert {:ok, output} = Runner.run(@media_url, [], "")
+4 -4
View File
@@ -5,7 +5,7 @@ defmodule Pinchflat.MediaFixtures do
""" """
alias Pinchflat.SourcesFixtures alias Pinchflat.SourcesFixtures
alias Pinchflat.Filesystem.FilesystemHelpers alias Pinchflat.Utils.FilesystemUtils
@doc """ @doc """
Generate a media_item. Generate a media_item.
@@ -52,8 +52,8 @@ defmodule Pinchflat.MediaFixtures do
json_gz_filepath = Path.join(metadata_dir, "metadata.json.gz") json_gz_filepath = Path.join(metadata_dir, "metadata.json.gz")
thumbnail_filepath = Path.join(metadata_dir, "thumbnail.jpg") thumbnail_filepath = Path.join(metadata_dir, "thumbnail.jpg")
FilesystemHelpers.cp_p!(media_metadata_filepath_fixture(), json_gz_filepath) FilesystemUtils.cp_p!(media_metadata_filepath_fixture(), json_gz_filepath)
FilesystemHelpers.cp_p!(thumbnail_filepath_fixture(), thumbnail_filepath) FilesystemUtils.cp_p!(thumbnail_filepath_fixture(), thumbnail_filepath)
merged_attrs = merged_attrs =
Map.merge(attrs, %{ Map.merge(attrs, %{
@@ -74,7 +74,7 @@ defmodule Pinchflat.MediaFixtures do
"#{:rand.uniform(1_000_000)}_media.mp4" "#{:rand.uniform(1_000_000)}_media.mp4"
]) ])
FilesystemHelpers.cp_p!(media_filepath_fixture(), stored_media_filepath) FilesystemUtils.cp_p!(media_filepath_fixture(), stored_media_filepath)
merged_attrs = Map.merge(attrs, %{media_filepath: stored_media_filepath}) merged_attrs = Map.merge(attrs, %{media_filepath: stored_media_filepath})
media_item_fixture(merged_attrs) media_item_fixture(merged_attrs)
+4 -4
View File
@@ -8,7 +8,7 @@ defmodule Pinchflat.SourcesFixtures do
alias Pinchflat.MediaFixtures alias Pinchflat.MediaFixtures
alias Pinchflat.Sources.Source alias Pinchflat.Sources.Source
alias Pinchflat.ProfilesFixtures alias Pinchflat.ProfilesFixtures
alias Pinchflat.Filesystem.FilesystemHelpers alias Pinchflat.Utils.FilesystemUtils
@doc """ @doc """
Generate a source. Generate a source.
@@ -59,9 +59,9 @@ defmodule Pinchflat.SourcesFixtures do
poster_filepath = Path.join(metadata_dir, "poster.jpg") poster_filepath = Path.join(metadata_dir, "poster.jpg")
fanart_filepath = Path.join(metadata_dir, "fanart.jpg") fanart_filepath = Path.join(metadata_dir, "fanart.jpg")
FilesystemHelpers.cp_p!(MediaFixtures.media_metadata_filepath_fixture(), json_gz_filepath) FilesystemUtils.cp_p!(MediaFixtures.media_metadata_filepath_fixture(), json_gz_filepath)
FilesystemHelpers.cp_p!(MediaFixtures.thumbnail_filepath_fixture(), poster_filepath) FilesystemUtils.cp_p!(MediaFixtures.thumbnail_filepath_fixture(), poster_filepath)
FilesystemHelpers.cp_p!(MediaFixtures.thumbnail_filepath_fixture(), fanart_filepath) FilesystemUtils.cp_p!(MediaFixtures.thumbnail_filepath_fixture(), fanart_filepath)
merged_attrs = merged_attrs =
Map.merge(attrs, %{ Map.merge(attrs, %{