Compare commits

...

20 Commits

Author SHA1 Message Date
Kieran Eglin 1b7fb6d14b Bumped version [skip ci] 2024-05-03 16:13:28 -07:00
Kieran 944d26b57d Added test button to apprise input (#226) 2024-05-03 15:42:22 -07:00
Kieran 5427c6247f [Enhancement] Add "Copy JSON" action button to help when troubleshooting (#224)
* [WIP] [skip ci]

* Added 'copy JSON' button to all dropdowns; hopefully fixed some CSS silliness
2024-05-03 10:07:59 -07:00
Kieran f655a8ae01 Improved robustness of file downloads (#225) 2024-05-03 09:15:12 -07:00
Kieran Eglin 7090349abd Bumped version 2024-05-02 12:13:14 -07:00
Kieran f2c9c10437 Added reload button to live tables (#223) 2024-05-02 12:09:55 -07:00
Kieran ff188f9351 Added UI to indicate a new version (#222) 2024-05-02 11:39:12 -07:00
Kieran 04b14719ee [Enhancement] Add Media Center support for videos uploaded on the same day (#221)
* Added upload date index field to media_items

* Added incrementing index for upload dates

* Added media item upload date index to download option builder

* Added new season_episode_index_from_date to UI; updated parser

* Improve support for channels

* Hopefully fixed flakey test
2024-05-02 11:06:10 -07:00
Kieran 112c6a4f14 [Enhancement] Custom media lifecycle scripts (#219)
* Namespaced notification modules under lifecycle

* Added a JSON encoder for all the main model types

* Added startup task to create user script file

* Hook up user script event to media download

* Hooked up media deletion user script

* Added jq to docker deps

* Updated README
2024-05-02 08:43:37 -07:00
Kieran 8051107d32 Stopped notifications from being sent if the source doesn't download (#218) 2024-05-01 08:56:15 -07:00
Kieran 3f74f199dc Stopped worker from retrying if doing so wouldn't improve things (#210) 2024-04-29 14:06:12 -07:00
Kieran 09cac46e14 Adds thumbnails as episode-level images for podcasts (#201) 2024-04-26 12:10:12 -07:00
Kieran Eglin f65ebea738 Bumped version 2024-04-25 11:07:04 -07:00
Kieran edb48b3989 Removed worker (#199) 2024-04-25 11:06:30 -07:00
Kieran 98c2812ee8 [Enhancement] Allow forcing a refresh of source metadata (#194)
* Stopped sources from fetching metadata on every update

* Added action button to force a metadata refresh
2024-04-18 15:35:53 -07:00
Kieran Eglin 526bc0c2e3 Moved podcast stuff out of beta 2024-04-17 14:54:35 -07:00
Kieran Eglin 5a5448b87a Improved README 2024-04-17 14:44:00 -07:00
Kieran 88569640c9 [Bugfix] Ensure source metadata fetching uses windows-compatible filepaths (#193)
* Fixed underlying bug

* Refactored windows_filenames to be a global flag; added tests

* Added some random test coverage for fun
2024-04-17 12:51:52 -07:00
Kieran 6f78ec40d7 [Enhancement] Improve layout of media item page on large displays (#192)
* Improved layout of media items page

* Removed useless anchor tag
2024-04-17 11:31:50 -07:00
Kieran aea40a3f92 [Enhancement] Add pagination for a source's media (#190)
* [WIP] added first attempt at pagination component

* Hooked up pagination for downloaded media
2024-04-17 10:22:55 -07:00
104 changed files with 1817 additions and 401 deletions
+26 -5
View File
@@ -12,6 +12,14 @@
</sup> </sup>
</p> </p>
<div align="center">
[![](https://img.shields.io/github/license/kieraneglin/pinchflat?style=for-the-badge&color=ee512b)](LICENSE)
[![](https://img.shields.io/github/v/release/kieraneglin/pinchflat?style=for-the-badge)](https://github.com/kieraneglin/pinchflat/releases)
[![](https://img.shields.io/github/actions/workflow/status/kieraneglin/pinchflat/lint_and_test.yml?style=for-the-badge)](#)
</div>
# Your next YouTube media manager # Your next YouTube media manager
## Table of contents: ## Table of contents:
@@ -21,6 +29,7 @@
- [Screenshots](#screenshots) - [Screenshots](#screenshots)
- [Installation](#installation) - [Installation](#installation)
- [Unraid](#unraid) - [Unraid](#unraid)
- [Portainer](#portainer)
- [Docker](#docker) - [Docker](#docker)
- [Username and Password (authentication)](https://github.com/kieraneglin/pinchflat/wiki/Username-and-Password) - [Username and Password (authentication)](https://github.com/kieraneglin/pinchflat/wiki/Username-and-Password)
- [Frequently asked questions](https://github.com/kieraneglin/pinchflat/wiki/Frequently-Asked-Questions) - [Frequently asked questions](https://github.com/kieraneglin/pinchflat/wiki/Frequently-Asked-Questions)
@@ -43,7 +52,7 @@ If it doesn't work for your use case, please make a feature request! You can als
- Powerful naming system so content is stored where and how you want it - Powerful naming system so content is stored where and how you want it
- Easy-to-use web interface with presets to get you started right away - Easy-to-use web interface with presets to get you started right away
- First-class support for media center apps like Plex, Jellyfin, and Kodi - First-class support for media center apps like Plex, Jellyfin, and Kodi
- Supports serving RSS feeds to your favourite podcast app (beta - [docs](<https://github.com/kieraneglin/pinchflat/wiki/Podcast-RSS-Feeds-(beta)>)) - Supports serving RSS feeds to your favourite podcast app ([docs](https://github.com/kieraneglin/pinchflat/wiki/Podcast-RSS-Feeds))
- Automatically downloads new content from channels and playlists - Automatically downloads new content from channels and playlists
- Uses a novel approach to download new content more quickly than other apps - Uses a novel approach to download new content more quickly than other apps
- Supports downloading audio content - Supports downloading audio content
@@ -57,6 +66,7 @@ If it doesn't work for your use case, please make a feature request! You can als
- Can pass cookies to YouTube to download your private playlists ([docs](https://github.com/kieraneglin/pinchflat/wiki/YouTube-Cookies)) - Can pass cookies to YouTube to download your private playlists ([docs](https://github.com/kieraneglin/pinchflat/wiki/YouTube-Cookies))
- Sponsorblock integration - Sponsorblock integration
- \[Advanced\] allows custom `yt-dlp` options ([docs](https://github.com/kieraneglin/pinchflat/wiki/%5BAdvanced%5D-Custom-yt%E2%80%90dlp-options)) - \[Advanced\] allows custom `yt-dlp` options ([docs](https://github.com/kieraneglin/pinchflat/wiki/%5BAdvanced%5D-Custom-yt%E2%80%90dlp-options))
- \[Advanced\] supports running custom scripts when after downloading/deleting media (alpha - [docs](https://github.com/kieraneglin/pinchflat/wiki/%5BAdvanced%5D-Custom-lifecycle-scripts))
## Screenshots ## Screenshots
@@ -71,7 +81,8 @@ Simply search for Pinchflat in the Community Apps store!
### Portainer ### Portainer
Important: See the note below about storing config on a network file share. It's preferred to store the config on a local disk if at all possible. > [!IMPORTANT]
> See the note below about storing config on a network file share. It's preferred to store the config on a local disk if at all possible.
Docker Compose file: Docker Compose file:
@@ -110,13 +121,18 @@ docker run \
You _must_ ensure the host directories you've mounted are writable by the user running the Docker container. If you get a permission error follow the steps it suggests. See [#106](https://github.com/kieraneglin/pinchflat/issues/106) for more. You _must_ ensure the host directories you've mounted are writable by the user running the Docker container. If you get a permission error follow the steps it suggests. See [#106](https://github.com/kieraneglin/pinchflat/issues/106) for more.
It's recommended to not run the container as root. Doing so can create permission issues if other apps need to work with the downloaded media. If you need to run any command as root, you can run `su` from the container's shell as there is no password set for the root user. > [!IMPORTANT]
> It's not recommended to run the container as root. Doing so can create permission issues if other apps need to work with the downloaded media.
### Advanced: storing Pinchflat config directory on a network share > [!TIP]
> If you need to run any command as root, you can run `su` from the container's shell as there is no password set for the root user.
### ADVANCED: Storing Pinchflat config directory on a network share
As pointed out in [#137](https://github.com/kieraneglin/pinchflat/issues/137), SQLite doesn't like being run in WAL mode on network shares. If you're running Pinchflat on a network share, you can disable WAL mode by setting the `JOURNAL_MODE` environment variable to `delete`. This will make Pinchflat run in rollback journal mode which is less performant but should work on network shares. As pointed out in [#137](https://github.com/kieraneglin/pinchflat/issues/137), SQLite doesn't like being run in WAL mode on network shares. If you're running Pinchflat on a network share, you can disable WAL mode by setting the `JOURNAL_MODE` environment variable to `delete`. This will make Pinchflat run in rollback journal mode which is less performant but should work on network shares.
Changing this setting from WAL to `delete` on an existing Pinchflat instance could, conceivably, result in data loss. Only change this setting if you know what you're doing, why this is important, and are okay with possible data loss or DB corruption. Backup your database first! > [!CAUTION]
> Changing this setting from WAL to `delete` on an existing Pinchflat instance could, conceivably, result in data loss. Only change this setting if you know what you're doing, why this is important, and are okay with possible data loss or DB corruption. Backup your database first!
If you change this setting and it works well for you, please leave a comment on [#137](https://github.com/kieraneglin/pinchflat/issues/137)! Doubly so if it does _not_ work well. If you change this setting and it works well for you, please leave a comment on [#137](https://github.com/kieraneglin/pinchflat/issues/137)! Doubly so if it does _not_ work well.
@@ -131,3 +147,8 @@ This is pre-release software and anything can break at any time. I make not guar
## License ## License
See `LICENSE` file See `LICENSE` file
<!-- Images and links -->
[license-badge]: https://img.shields.io/github/license/kieraneglin/pinchflat?style=for-the-badge&color=ee512b
[license-badge-url]: LICENSE
+37
View File
@@ -0,0 +1,37 @@
window.copyTextToClipboard = async (text) => {
// Navigator clipboard api needs a secure context (https)
if (navigator.clipboard && window.isSecureContext) {
await navigator.clipboard.writeText(text)
} else {
const textArea = document.createElement('textarea')
textArea.value = text
// Move textarea out of the viewport so it's not visible
textArea.style.position = 'absolute'
textArea.style.left = '-999999px'
document.body.prepend(textArea)
textArea.select()
try {
document.execCommand('copy')
} catch (error) {
console.error(error)
} finally {
textArea.remove()
}
}
}
window.copyWithCallbacks = async (text, onCopy, onAfterDelay, delay = 4000) => {
await window.copyTextToClipboard(text)
onCopy()
setTimeout(onAfterDelay, delay)
}
window.markVersionAsSeen = (versionString) => {
localStorage.setItem('seenVersion', versionString)
}
window.isVersionSeen = (versionString) => {
return localStorage.getItem('seenVersion') === versionString
}
+1 -24
View File
@@ -23,6 +23,7 @@ import { LiveSocket } from 'phoenix_live_view'
import topbar from '../vendor/topbar' import topbar from '../vendor/topbar'
import Alpine from 'alpinejs' import Alpine from 'alpinejs'
import './tabs' import './tabs'
import './alpine_helpers'
window.Alpine = Alpine window.Alpine = Alpine
Alpine.start() Alpine.start()
@@ -50,30 +51,6 @@ let liveSocket = new LiveSocket('/live', Socket, {
} }
}) })
window.copyTextToClipboard = async (text) => {
// Navigator clipboard api needs a secure context (https)
if (navigator.clipboard && window.isSecureContext) {
await navigator.clipboard.writeText(text)
} else {
const textArea = document.createElement('textarea')
textArea.value = text
// Move textarea out of the viewport so it's not visible
textArea.style.position = 'absolute'
textArea.style.left = '-999999px'
document.body.prepend(textArea)
textArea.select()
try {
document.execCommand('copy')
} catch (error) {
console.error(error)
} finally {
textArea.remove()
}
}
}
// Show progress bar on live navigation and form submits // Show progress bar on live navigation and form submits
topbar.config({ barColors: { 0: '#29d' }, shadowColor: 'rgba(0, 0, 0, .3)' }) topbar.config({ barColors: { 0: '#29d' }, shadowColor: 'rgba(0, 0, 0, .3)' })
window.addEventListener('phx:page-loading-start', (_info) => topbar.show(300)) window.addEventListener('phx:page-loading-start', (_info) => topbar.show(300))
+1 -1
View File
@@ -14,7 +14,7 @@ config :pinchflat,
yt_dlp_executable: System.find_executable("yt-dlp"), yt_dlp_executable: System.find_executable("yt-dlp"),
apprise_executable: System.find_executable("apprise"), apprise_executable: System.find_executable("apprise"),
yt_dlp_runner: Pinchflat.YtDlp.CommandRunner, yt_dlp_runner: Pinchflat.YtDlp.CommandRunner,
apprise_runner: Pinchflat.Notifications.CommandRunner, apprise_runner: Pinchflat.Lifecycle.Notifications.CommandRunner,
media_directory: "/downloads", media_directory: "/downloads",
# The user may or may not store metadata for their needs, but the app will always store its copy # The user may or may not store metadata for their needs, but the app will always store its copy
metadata_directory: "/config/metadata", metadata_directory: "/config/metadata",
+1 -1
View File
@@ -7,7 +7,7 @@ FROM ${DEV_IMAGE}
# Install debian packages # Install debian packages
RUN apt-get update -qq RUN apt-get update -qq
RUN apt-get install -y inotify-tools ffmpeg curl git openssh-client \ RUN apt-get install -y inotify-tools ffmpeg curl git openssh-client jq \
python3 python3-pip python3-setuptools python3-wheel python3-dev locales procps python3 python3-pip python3-setuptools python3-wheel python3-dev locales procps
# Install nodejs # Install nodejs
-70
View File
@@ -1,70 +0,0 @@
defmodule Pinchflat.Boot.NfoBackfillWorker do
@moduledoc false
use Oban.Worker,
queue: :local_metadata,
# This should have it running once _ever_ (until the job is pruned, anyway)
# NOTE: remove within the next month
unique: [period: :infinity, states: Oban.Job.states()],
tags: ["media_item", "media_metadata", "local_metadata", "data_backfill"]
import Ecto.Query, warn: false
require Logger
alias Pinchflat.Repo
alias Pinchflat.Media
alias Pinchflat.Media.MediaItem
alias Pinchflat.Metadata.NfoBuilder
alias Pinchflat.Metadata.MetadataFileHelpers
@doc """
Runs a one-off backfill job to regenerate NFO files for media items that have
both an NFO file and a metadata file. This is needed because NFO files weren't
escaping characters properly so we need to regenerate them.
This job will only run once as long as I remove it before the jobs are pruned in a month.
Returns :ok
"""
@impl Oban.Worker
def perform(%Oban.Job{}) do
Logger.info("Running NFO backfill worker")
media_items = get_media_items_to_backfill()
Enum.each(media_items, fn media_item ->
nfo_exists = File.exists?(media_item.nfo_filepath)
metadata_exists = File.exists?(media_item.metadata.metadata_filepath)
if nfo_exists && metadata_exists do
Logger.info("NFO and metadata exist for media item #{media_item.id} - proceeding")
regenerate_nfo_for_media_item(media_item)
end
end)
:ok
end
defp get_media_items_to_backfill do
from(m in MediaItem, where: not is_nil(m.nfo_filepath))
|> Repo.all()
|> Repo.preload([:metadata, source: :media_profile])
end
defp regenerate_nfo_for_media_item(media_item) do
try do
case MetadataFileHelpers.read_compressed_metadata(media_item.metadata.metadata_filepath) do
{:ok, metadata} ->
Media.update_media_item(media_item, %{
nfo_filepath: NfoBuilder.build_and_store_for_media_item(media_item.nfo_filepath, metadata)
})
_err ->
Logger.error("Failed to read metadata for media item #{media_item.id}")
end
rescue
e -> Logger.error("Unknown error regenerating NFO file for MI ##{media_item.id}: #{inspect(e)}")
end
end
end
+1 -4
View File
@@ -7,9 +7,6 @@ defmodule Pinchflat.Boot.PostJobStartupTasks do
Phoenix supervision tree. Phoenix supervision tree.
""" """
alias Pinchflat.Repo
alias Pinchflat.Boot.NfoBackfillWorker
# restart: :temporary means that this process will never be restarted (ie: will run once and then die) # restart: :temporary means that this process will never be restarted (ie: will run once and then die)
use GenServer, restart: :temporary use GenServer, restart: :temporary
import Ecto.Query, warn: false import Ecto.Query, warn: false
@@ -29,7 +26,7 @@ defmodule Pinchflat.Boot.PostJobStartupTasks do
""" """
@impl true @impl true
def init(state) do def init(state) do
Repo.insert_unique_job(NfoBackfillWorker.new(%{})) # Nothing at the moment!
{:ok, state} {:ok, state}
end end
@@ -33,6 +33,7 @@ defmodule Pinchflat.Boot.PreJobStartupTasks do
def init(state) do def init(state) do
reset_executing_jobs() reset_executing_jobs()
create_blank_yt_dlp_files() create_blank_yt_dlp_files()
create_blank_user_script_file()
apply_default_settings() apply_default_settings()
{:ok, state} {:ok, state}
@@ -65,6 +66,18 @@ defmodule Pinchflat.Boot.PreJobStartupTasks do
end) end)
end end
defp create_blank_user_script_file do
base_dir = Application.get_env(:pinchflat, :extras_directory)
filepath = Path.join([base_dir, "user-scripts", "lifecycle"])
if !File.exists?(filepath) do
Logger.info("Creating blank file and making it executable: #{filepath}")
FilesystemUtils.write_p!(filepath, "")
File.chmod(filepath, 0o755)
end
end
defp apply_default_settings do defp apply_default_settings do
{:ok, yt_dlp_version} = yt_dlp_runner().version() {:ok, yt_dlp_version} = yt_dlp_runner().version()
{:ok, apprise_version} = apprise_runner().version() {:ok, apprise_version} = apprise_runner().version()
@@ -8,6 +8,8 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilder do
alias Pinchflat.Media.MediaItem alias Pinchflat.Media.MediaItem
alias Pinchflat.Downloading.OutputPathBuilder alias Pinchflat.Downloading.OutputPathBuilder
alias Pinchflat.Utils.FilesystemUtils, as: FSUtils
@doc """ @doc """
Builds the options for yt-dlp to download media based on the given media's profile. Builds the options for yt-dlp to download media based on the given media's profile.
@@ -33,18 +35,24 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilder do
Builds the output path for yt-dlp to download media based on the given source's Builds the output path for yt-dlp to download media based on the given source's
media profile. Uses the source's override output path template if it exists. media profile. Uses the source's override output path template if it exists.
Accepts a %MediaItem{} or %Source{} struct. If a %Source{} struct is passed, it
will use a default %MediaItem{} struct with the given source.
Returns binary() Returns binary()
""" """
def build_output_path_for(%Source{} = source_with_preloads) do def build_output_path_for(%MediaItem{} = media_item_with_preloads) do
output_path_template = Sources.output_path_template(source_with_preloads) output_path_template = Sources.output_path_template(media_item_with_preloads.source)
build_output_path(output_path_template, source_with_preloads) build_output_path(output_path_template, media_item_with_preloads)
end
def build_output_path_for(%Source{} = source_with_preloads) do
build_output_path_for(%MediaItem{source: source_with_preloads})
end end
defp default_options do defp default_options do
[ [
:no_progress, :no_progress,
:windows_filenames,
# Add force-overwrites to make sure redownloading works # Add force-overwrites to make sure redownloading works
:force_overwrites, :force_overwrites,
# This makes the date metadata conform to what jellyfin expects # This makes the date metadata conform to what jellyfin expects
@@ -116,9 +124,18 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilder do
[format_sort: "res:#{res},+codec:avc:m4a", remux_video: "mp4"] [format_sort: "res:#{res},+codec:avc:m4a", remux_video: "mp4"]
end end
audio_format_precedence = [
"bestaudio[ext=m4a]",
"bestaudio[ext=mp3]",
"bestaudio",
"best[ext=m4a]",
"best[ext=mp3]",
"best"
]
case media_profile.preferred_resolution do case media_profile.preferred_resolution do
# Also be aware that :audio disabled all embedding options for subtitles # Also be aware that :audio disabled all embedding options for subtitles
:audio -> [:extract_audio, format: "bestaudio[ext=m4a]"] :audio -> [:extract_audio, format: Enum.join(audio_format_precedence, "/")]
:"360p" -> video_codec_option.("360") :"360p" -> video_codec_option.("360")
:"480p" -> video_codec_option.("480") :"480p" -> video_codec_option.("480")
:"720p" -> video_codec_option.("720") :"720p" -> video_codec_option.("720")
@@ -155,12 +172,10 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilder do
Enum.reduce(filenames, [], fn filename, acc -> Enum.reduce(filenames, [], fn filename, acc ->
filepath = Path.join(base_dir, filename) filepath = Path.join(base_dir, filename)
case File.read(filepath) do if FSUtils.exists_and_nonempty?(filepath) do
{:ok, file_data} -> [filepath | acc]
if String.trim(file_data) != "", do: [filepath | acc], else: acc else
acc
{:error, _} ->
acc
end end
end) end)
@@ -169,23 +184,29 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilder do
defp output_options(media_item_with_preloads) do defp output_options(media_item_with_preloads) do
[ [
output: build_output_path_for(media_item_with_preloads.source) output: build_output_path_for(media_item_with_preloads)
] ]
end end
defp build_output_path(string, source) do defp build_output_path(string, media_item_with_preloads) do
additional_options_map = output_options_map(source) additional_options_map = output_options_map(media_item_with_preloads)
{:ok, output_path} = OutputPathBuilder.build(string, additional_options_map) {:ok, output_path} = OutputPathBuilder.build(string, additional_options_map)
Path.join(base_directory(), output_path) Path.join(base_directory(), output_path)
end end
defp output_options_map(source) do defp output_options_map(media_item_with_preloads) do
source = media_item_with_preloads.source
%{ %{
"source_custom_name" => source.custom_name, "source_custom_name" => source.custom_name,
"source_collection_id" => source.collection_id, "source_collection_id" => source.collection_id,
"source_collection_name" => source.collection_name, "source_collection_name" => source.collection_name,
"source_collection_type" => source.collection_type "source_collection_type" => to_string(source.collection_type),
"media_upload_date_index" =>
media_item_with_preloads.upload_date_index
|> to_string()
|> String.pad_leading(2, "0")
} }
end end
@@ -199,7 +220,7 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilder do
|> String.split(~r{\.}, include_captures: true) |> String.split(~r{\.}, include_captures: true)
|> List.insert_at(-3, "-thumb") |> List.insert_at(-3, "-thumb")
|> Enum.join() |> Enum.join()
|> build_output_path(media_item_with_preloads.source) |> build_output_path(media_item_with_preloads)
end end
defp base_directory do defp base_directory do
@@ -14,6 +14,8 @@ defmodule Pinchflat.Downloading.MediaDownloadWorker do
alias Pinchflat.Media alias Pinchflat.Media
alias Pinchflat.Downloading.MediaDownloader alias Pinchflat.Downloading.MediaDownloader
alias Pinchflat.Lifecycle.UserScripts.CommandRunner, as: UserScriptRunner
@doc """ @doc """
Starts the media_item media download worker and creates a task for the media_item. Starts the media_item media download worker and creates a task for the media_item.
@@ -56,37 +58,54 @@ defmodule Pinchflat.Downloading.MediaDownloadWorker do
defp download_media_and_schedule_jobs(media_item, is_redownload) do defp download_media_and_schedule_jobs(media_item, is_redownload) do
case MediaDownloader.download_for_media_item(media_item) do case MediaDownloader.download_for_media_item(media_item) do
{:ok, updated_media_item} -> {:ok, downloaded_media_item} ->
Media.update_media_item(updated_media_item, %{ {:ok, updated_media_item} =
media_size_bytes: compute_media_filesize(updated_media_item), Media.update_media_item(downloaded_media_item, %{
media_redownloaded_at: get_redownloaded_at(is_redownload) media_size_bytes: compute_media_filesize(downloaded_media_item),
}) media_redownloaded_at: get_redownloaded_at(is_redownload)
})
:ok = run_user_script(updated_media_item)
{:ok, updated_media_item} {:ok, updated_media_item}
{:recovered, _} -> {:recovered, _} ->
{:error, :retry} {:error, :retry}
{:error, _message} -> {:error, message} ->
{:error, :download_failed} action_on_error(message)
end end
end end
defp compute_media_filesize(media_item) do defp compute_media_filesize(media_item) do
case File.stat(media_item.media_filepath) do case File.stat(media_item.media_filepath) do
{:ok, %{size: size}} -> {:ok, %{size: size}} -> size
size _ -> nil
_ ->
nil
end end
end end
defp get_redownloaded_at(is_redownload) do defp get_redownloaded_at(true), do: DateTime.utc_now()
if is_redownload do defp get_redownloaded_at(_), do: nil
DateTime.utc_now()
defp action_on_error(message) do
# This will attempt re-download at the next indexing, but it won't be retried
# immediately as part of job failure logic
non_retryable_errors = ["Video unavailable"]
if String.contains?(to_string(message), non_retryable_errors) do
Logger.error("yt-dlp download will not be retried: #{inspect(message)}")
{:ok, :non_retry}
else else
nil {:error, :download_failed}
end end
end end
# NOTE: I like this pattern of using the default value so that I don't have to
# define it in config.exs (and friends). Consider using this elsewhere.
defp run_user_script(media_item) do
runner = Application.get_env(:pinchflat, :user_script_runner, UserScriptRunner)
runner.run(:media_downloaded, media_item)
end
end end
@@ -9,13 +9,23 @@ defmodule Pinchflat.Downloading.OutputPathBuilder do
Builds the actual final filepath from a given template. Optionally, you can pass in Builds the actual final filepath from a given template. Optionally, you can pass in
a map of additional options to be used in the template. a map of additional options to be used in the template.
Custom options are recursively expanded _once_ so you can nest custom options
one-deep if needed.
Translates liquid-style templates into yt-dlp-style templates, Translates liquid-style templates into yt-dlp-style templates,
leaving yt-dlp syntax intact. leaving yt-dlp syntax intact.
""" """
def build(template_string, additional_template_options \\ %{}) do def build(template_string, additional_template_options \\ %{}) do
combined_options = Map.merge(custom_yt_dlp_option_map(), additional_template_options) combined_options = Map.merge(custom_yt_dlp_option_map(), additional_template_options)
TemplateParser.parse(template_string, combined_options, &identifier_fn/2) expanded_options =
Enum.map(combined_options, fn {key, value} ->
{:ok, parse_result} = TemplateParser.parse(value, combined_options, &identifier_fn/2)
{key, parse_result}
end)
TemplateParser.parse(template_string, Map.new(expanded_options), &identifier_fn/2)
end end
# The `nil` case simply wraps the identifier in yt-dlp-style syntax. This assumes that # The `nil` case simply wraps the identifier in yt-dlp-style syntax. This assumes that
@@ -43,6 +53,7 @@ defmodule Pinchflat.Downloading.OutputPathBuilder do
"upload_yyyy_mm_dd" => "%(upload_date>%Y-%m-%d)S", "upload_yyyy_mm_dd" => "%(upload_date>%Y-%m-%d)S",
"season_from_date" => "%(upload_date>%Y)S", "season_from_date" => "%(upload_date>%Y)S",
"season_episode_from_date" => "s%(upload_date>%Y)Se%(upload_date>%m%d)S", "season_episode_from_date" => "s%(upload_date>%Y)Se%(upload_date>%m%d)S",
"season_episode_index_from_date" => "s%(upload_date>%Y)Se%(upload_date>%m%d)S{{ media_upload_date_index }}",
"artist_name" => "%(artist,creator,uploader,uploader_id)S" "artist_name" => "%(artist,creator,uploader,uploader_id)S"
} }
end end
@@ -15,7 +15,7 @@ defmodule Pinchflat.FastIndexing.FastIndexingWorker do
alias Pinchflat.Settings alias Pinchflat.Settings
alias Pinchflat.Sources.Source alias Pinchflat.Sources.Source
alias Pinchflat.FastIndexing.FastIndexingHelpers alias Pinchflat.FastIndexing.FastIndexingHelpers
alias Pinchflat.Notifications.SourceNotifications alias Pinchflat.Lifecycle.Notifications.SourceNotifications
@doc """ @doc """
Starts the source fast indexing worker and creates a task for the source. Starts the source fast indexing worker and creates a task for the source.
@@ -1,4 +1,4 @@
defmodule Pinchflat.Notifications.AppriseCommandRunner do defmodule Pinchflat.Lifecycle.Notifications.AppriseCommandRunner do
@moduledoc """ @moduledoc """
A behaviour for running CLI commands against a notification backend (apprise). A behaviour for running CLI commands against a notification backend (apprise).
@@ -1,4 +1,4 @@
defmodule Pinchflat.Notifications.CommandRunner do defmodule Pinchflat.Lifecycle.Notifications.CommandRunner do
@moduledoc """ @moduledoc """
Runs apprise commands using the `System.cmd/3` function Runs apprise commands using the `System.cmd/3` function
""" """
@@ -7,7 +7,7 @@ defmodule Pinchflat.Notifications.CommandRunner do
alias Pinchflat.Utils.CliUtils alias Pinchflat.Utils.CliUtils
alias Pinchflat.Utils.FunctionUtils alias Pinchflat.Utils.FunctionUtils
alias Pinchflat.Notifications.AppriseCommandRunner alias Pinchflat.Lifecycle.Notifications.AppriseCommandRunner
@behaviour AppriseCommandRunner @behaviour AppriseCommandRunner
@@ -28,11 +28,10 @@ defmodule Pinchflat.Notifications.CommandRunner do
default_opts = [:verbose] default_opts = [:verbose]
parsed_opts = CliUtils.parse_options(default_opts ++ command_opts) parsed_opts = CliUtils.parse_options(default_opts ++ command_opts)
Logger.info("[apprise] called with: #{Enum.join(parsed_opts ++ endpoints, " ")}") {output, exit_code} = CliUtils.wrap_cmd(backend_executable(), parsed_opts ++ endpoints)
{output, return_code} = System.cmd(backend_executable(), parsed_opts ++ endpoints)
Logger.info("[apprise] response: #{output}") Logger.info("[apprise] response: #{output}")
case return_code do case exit_code do
0 -> {:ok, String.trim(output)} 0 -> {:ok, String.trim(output)}
_ -> {:error, String.trim(output)} _ -> {:error, String.trim(output)}
end end
@@ -1,4 +1,4 @@
defmodule Pinchflat.Notifications.SourceNotifications do defmodule Pinchflat.Lifecycle.Notifications.SourceNotifications do
@moduledoc """ @moduledoc """
Contains utilities for sending notifications about sources Contains utilities for sending notifications about sources
""" """
@@ -53,7 +53,11 @@ defmodule Pinchflat.Notifications.SourceNotifications do
end end
defp relevant_media_item_count(source) do defp relevant_media_item_count(source) do
pending_media_item_count(source) + downloaded_media_item_count(source) if source.download_media do
pending_media_item_count(source) + downloaded_media_item_count(source)
else
0
end
end end
defp pending_media_item_count(source) do defp pending_media_item_count(source) do
@@ -0,0 +1,76 @@
defmodule Pinchflat.Lifecycle.UserScripts.CommandRunner do
@moduledoc """
Runs custom user commands commands using the `System.cmd/3` function
"""
require Logger
alias Pinchflat.Utils.CliUtils
alias Pinchflat.Utils.FilesystemUtils
alias Pinchflat.Lifecycle.UserScripts.UserScriptCommandRunner
@behaviour UserScriptCommandRunner
@event_types [
:media_downloaded,
:media_deleted
]
@doc """
Runs the user script command for the given event type. Passes the event
and the encoded data to the user script command.
This function will succeed in almost all cases, even if the user script command
failed - this is because I don't want bad scripts to stop the whole process.
If something fails, it'll be logged.
The only things that can cause a true failure are passing in an invalid event
type or if the passed data cannot be encoded into JSON - both indicative of
failures in the development process.
Returns :ok
"""
@impl UserScriptCommandRunner
def run(event_type, encodable_data) when event_type in @event_types do
case backend_executable() do
{:ok, :no_executable} ->
:ok
{:ok, executable_path} ->
{:ok, encoded_data} = Phoenix.json_library().encode(encodable_data)
{output, exit_code} =
CliUtils.wrap_cmd(
executable_path,
[to_string(event_type), encoded_data],
[],
logging_arg_override: "[suppressed]"
)
handle_output(output, exit_code)
end
end
def run(event_type, _encodable_data) do
raise ArgumentError, "Invalid event type: #{inspect(event_type)}"
end
defp handle_output(output, exit_code) do
Logger.debug("Custom lifecycle script exit code: #{exit_code} with output: #{output}")
:ok
end
defp backend_executable do
base_dir = Application.get_env(:pinchflat, :extras_directory)
filepath = Path.join([base_dir, "user-scripts", "lifecycle"])
if FilesystemUtils.exists_and_nonempty?(filepath) do
{:ok, filepath}
else
Logger.warning("User scripts lifecyle file either not present or is empty. Skipping.")
{:ok, :no_executable}
end
end
end
@@ -0,0 +1,10 @@
defmodule Pinchflat.Lifecycle.UserScripts.UserScriptCommandRunner do
@moduledoc """
A behaviour for running custom user scripts on certain events.
Used so we can implement Mox for testing without actually running the
user's command.
"""
@callback run(atom(), map()) :: :ok | {:error, binary()}
end
+11 -1
View File
@@ -10,8 +10,10 @@ defmodule Pinchflat.Media do
alias Pinchflat.Sources.Source alias Pinchflat.Sources.Source
alias Pinchflat.Media.MediaItem alias Pinchflat.Media.MediaItem
alias Pinchflat.Media.MediaQuery alias Pinchflat.Media.MediaQuery
alias Pinchflat.Metadata.MediaMetadata
alias Pinchflat.Utils.FilesystemUtils alias Pinchflat.Utils.FilesystemUtils
alias Pinchflat.Metadata.MediaMetadata
alias Pinchflat.Lifecycle.UserScripts.CommandRunner, as: UserScriptRunner
@doc """ @doc """
Returns the list of media_items. Returns the list of media_items.
@@ -180,6 +182,7 @@ defmodule Pinchflat.Media do
if delete_files do if delete_files do
{:ok, _} = do_delete_media_files(media_item) {:ok, _} = do_delete_media_files(media_item)
:ok = run_user_script(:media_deleted, media_item)
end end
# Should delete these no matter what # Should delete these no matter what
@@ -202,6 +205,7 @@ defmodule Pinchflat.Media do
Tasks.delete_tasks_for(media_item) Tasks.delete_tasks_for(media_item)
{:ok, _} = do_delete_media_files(media_item) {:ok, _} = do_delete_media_files(media_item)
:ok = run_user_script(:media_deleted, media_item)
update_media_item(media_item, Map.merge(filepath_attrs, addl_attrs)) update_media_item(media_item, Map.merge(filepath_attrs, addl_attrs))
end end
@@ -237,4 +241,10 @@ defmodule Pinchflat.Media do
|> Enum.filter(&is_binary/1) |> Enum.filter(&is_binary/1)
|> Enum.each(&FilesystemUtils.delete_file_and_remove_empty_directories/1) |> Enum.each(&FilesystemUtils.delete_file_and_remove_empty_directories/1)
end end
defp run_user_script(event, media_item) do
runner = Application.get_env(:pinchflat, :user_script_runner, UserScriptRunner)
runner.run(event, media_item)
end
end end
+45
View File
@@ -7,8 +7,12 @@ defmodule Pinchflat.Media.MediaItem do
import Ecto.Changeset import Ecto.Changeset
import Pinchflat.Utils.ChangesetUtils import Pinchflat.Utils.ChangesetUtils
alias __MODULE__
alias Pinchflat.Repo
alias Pinchflat.Sources
alias Pinchflat.Tasks.Task alias Pinchflat.Tasks.Task
alias Pinchflat.Sources.Source alias Pinchflat.Sources.Source
alias Pinchflat.Media.MediaQuery
alias Pinchflat.Metadata.MediaMetadata alias Pinchflat.Metadata.MediaMetadata
alias Pinchflat.Media.MediaItemsSearchIndex alias Pinchflat.Media.MediaItemsSearchIndex
@@ -22,6 +26,7 @@ defmodule Pinchflat.Media.MediaItem do
:source_id, :source_id,
:short_form_content, :short_form_content,
:upload_date, :upload_date,
:upload_date_index,
:duration_seconds, :duration_seconds,
# these fields are captured only on download # these fields are captured only on download
:media_downloaded_at, :media_downloaded_at,
@@ -64,6 +69,7 @@ defmodule Pinchflat.Media.MediaItem do
field :media_downloaded_at, :utc_datetime field :media_downloaded_at, :utc_datetime
field :media_redownloaded_at, :utc_datetime field :media_redownloaded_at, :utc_datetime
field :upload_date, :date field :upload_date, :date
field :upload_date_index, :integer, default: 0
field :duration_seconds, :integer field :duration_seconds, :integer
field :media_filepath, :string field :media_filepath, :string
@@ -98,6 +104,7 @@ defmodule Pinchflat.Media.MediaItem do
|> cast(attrs, @allowed_fields) |> cast(attrs, @allowed_fields)
|> cast_assoc(:metadata, with: &MediaMetadata.changeset/2, required: false) |> cast_assoc(:metadata, with: &MediaMetadata.changeset/2, required: false)
|> dynamic_default(:uuid, fn _ -> Ecto.UUID.generate() end) |> dynamic_default(:uuid, fn _ -> Ecto.UUID.generate() end)
|> update_upload_date_index()
|> validate_required(@required_fields) |> validate_required(@required_fields)
|> unique_constraint([:media_id, :source_id]) |> unique_constraint([:media_id, :source_id])
end end
@@ -116,4 +123,42 @@ defmodule Pinchflat.Media.MediaItem do
end) end)
|> Enum.into(%{}) |> Enum.into(%{})
end end
@doc false
def json_exluded_fields do
~w(__meta__ __struct__ metadata tasks media_items_search_index)a
end
defp update_upload_date_index(%{changes: changes} = changeset) when is_map_key(changes, :upload_date) do
source_id = get_field(changeset, :source_id)
source = Sources.get_source!(source_id)
# Channels should count down from 99, playlists should count up from 0
# This reflects the fact that channels prepend new videos to the top of the list
# and playlists append new videos to the bottom of the list.
default_index = if source.collection_type == :channel, do: 99, else: 0
aggregator = if source.collection_type == :channel, do: :min, else: :max
change_direction = if source.collection_type == :channel, do: -1, else: 1
current_max =
MediaQuery.new()
|> MediaQuery.for_source(source_id)
|> MediaQuery.where_uploaded_on_date(changes.upload_date)
|> Repo.aggregate(aggregator, :upload_date_index)
case current_max do
nil -> put_change(changeset, :upload_date_index, default_index)
max -> put_change(changeset, :upload_date_index, max + change_direction)
end
end
defp update_upload_date_index(changeset), do: changeset
defimpl Jason.Encoder, for: MediaItem do
def encode(value, opts) do
value
|> Repo.preload(:source)
|> Map.drop(MediaItem.json_exluded_fields())
|> Jason.Encode.map(opts)
end
end
end end
+8
View File
@@ -25,6 +25,10 @@ defmodule Pinchflat.Media.MediaQuery do
MediaItem MediaItem
end end
def for_source(query, source_id) when is_integer(source_id) do
where(query, [mi], mi.source_id == ^source_id)
end
def for_source(query, source) do def for_source(query, source) do
where(query, [mi], mi.source_id == ^source.id) where(query, [mi], mi.source_id == ^source.id)
end end
@@ -99,6 +103,10 @@ defmodule Pinchflat.Media.MediaQuery do
|> where([mi, source], is_nil(source.download_cutoff_date) or mi.upload_date >= source.download_cutoff_date) |> where([mi, source], is_nil(source.download_cutoff_date) or mi.upload_date >= source.download_cutoff_date)
end end
def where_uploaded_on_date(query, date) do
where(query, [mi], mi.upload_date == ^date)
end
def where_download_not_prevented(query) do def where_download_not_prevented(query) do
where(query, [mi], mi.prevent_download == false) where(query, [mi], mi.prevent_download == false)
end end
+24 -12
View File
@@ -54,24 +54,36 @@ defmodule Pinchflat.Metadata.MetadataFileHelpers do
@doc """ @doc """
Downloads and stores a thumbnail for a media item, returning the filepath. Downloads and stores a thumbnail for a media item, returning the filepath.
Chooses the highest quality jpg thumbnail available. Chooses the highest quality thumbnail available (preferring jpg). Returns
nil if no thumbnails are available.
Returns binary() Returns binary() | nil
""" """
def download_and_store_thumbnail_for(database_record, metadata_map) do def download_and_store_thumbnail_for(database_record, metadata_map) do
thumbnail_url = thumbnails =
metadata_map["thumbnails"] (metadata_map["thumbnails"] || [])
|> Enum.filter(&(&1["preference"] && String.ends_with?(&1["url"], ".jpg"))) # Give it a low preference if the `preference` key doesn't exist
|> Enum.sort(&(&1["preference"] >= &2["preference"])) |> Enum.map(&Map.put_new(&1, "preference", -1000))
|> List.first() # Give it a low preference if image isn't a jpg
|> Map.get("url") |> Enum.map(fn t ->
preference_weight = if String.ends_with?(t["url"], ".jpg"), do: t["preference"], else: t["preference"] - 1000
filepath = generate_filepath_for(database_record, Path.basename(thumbnail_url)) Map.put(t, "preference", preference_weight)
thumbnail_blob = fetch_thumbnail_from_url(thumbnail_url) end)
:ok = FilesystemUtils.write_p!(filepath, thumbnail_blob) case Enum.sort_by(thumbnails, & &1["preference"], :desc) do
[thumbnail_map | _] ->
thumbnail_url = thumbnail_map["url"]
filepath = generate_filepath_for(database_record, Path.basename(thumbnail_url))
thumbnail_blob = fetch_thumbnail_from_url(thumbnail_url)
filepath :ok = FilesystemUtils.write_p!(filepath, thumbnail_blob)
filepath
_ ->
nil
end
end end
@doc """ @doc """
+1 -1
View File
@@ -30,7 +30,7 @@ defmodule Pinchflat.Metadata.MetadataParser do
original_url: metadata["original_url"], original_url: metadata["original_url"],
description: metadata["description"], description: metadata["description"],
media_filepath: metadata["filepath"], media_filepath: metadata["filepath"],
livestream: metadata["was_live"], livestream: !!metadata["was_live"],
duration_seconds: metadata["duration"] && round(metadata["duration"]) duration_seconds: metadata["duration"] && round(metadata["duration"])
} }
end end
@@ -38,9 +38,9 @@ defmodule Pinchflat.Metadata.SourceMetadataStorageWorker do
- The NFO file for the source (if specified) - The NFO file for the source (if specified)
- Downloads and stores source images (if specified) - Downloads and stores source images (if specified)
The worker is kicked off after a source is inserted/updated - this can The worker is kicked off after a source is inserted or it's original_url
take an unknown amount of time so don't rely on this data being here is updated - this can take an unknown amount of time so don't rely on this
before, say, the first indexing or downloading task is complete. data being here before, say, the first indexing or downloading task is complete.
Returns :ok Returns :ok
""" """
@@ -75,6 +75,8 @@ defmodule Pinchflat.Podcasts.RssFeedBuilder do
end end
defp build_media_item_xml(source, media_item, url_base) do defp build_media_item_xml(source, media_item, url_base) do
item_image_path = item_image_path(url_base, media_item)
""" """
<item> <item>
<guid isPermaLink="false">#{media_item.uuid}</guid> <guid isPermaLink="false">#{media_item.uuid}</guid>
@@ -91,6 +93,10 @@ defmodule Pinchflat.Podcasts.RssFeedBuilder do
<itunes:author>#{safe(source.custom_name)}</itunes:author> <itunes:author>#{safe(source.custom_name)}</itunes:author>
<itunes:subtitle>#{safe(media_item.title)}</itunes:subtitle> <itunes:subtitle>#{safe(media_item.title)}</itunes:subtitle>
<itunes:summary><![CDATA[#{media_item.description}]]></itunes:summary> <itunes:summary><![CDATA[#{media_item.description}]]></itunes:summary>
#{item_image_path && ~s(<itunes:image href="#{safe(item_image_path)}"></itunes:image>)}
#{item_image_path && ~s(<podcast:images srcset="#{safe(item_image_path)}" />)}
<itunes:explicit>false</itunes:explicit> <itunes:explicit>false</itunes:explicit>
</item> </item>
""" """
@@ -117,6 +123,16 @@ defmodule Pinchflat.Podcasts.RssFeedBuilder do
end end
end end
def item_image_path(url_base, media_item) do
if media_item.thumbnail_filepath && File.exists?(media_item.thumbnail_filepath) do
extension = Path.extname(media_item.thumbnail_filepath)
Path.join(url_base, "#{podcast_route(:episode_image, media_item.uuid)}#{extension}")
else
nil
end
end
defp generate_upload_date(media_item) do defp generate_upload_date(media_item) do
media_item.upload_date media_item.upload_date
|> DatetimeUtils.date_to_datetime() |> DatetimeUtils.date_to_datetime()
+14
View File
@@ -6,6 +6,7 @@ defmodule Pinchflat.Profiles.MediaProfile do
use Ecto.Schema use Ecto.Schema
import Ecto.Changeset import Ecto.Changeset
alias __MODULE__
alias Pinchflat.Sources.Source alias Pinchflat.Sources.Source
@allowed_fields ~w( @allowed_fields ~w(
@@ -84,4 +85,17 @@ defmodule Pinchflat.Profiles.MediaProfile do
def ext_regex do def ext_regex do
~r/\.({{ ?ext ?}}|%\( ?ext ?\)[sS])$/ ~r/\.({{ ?ext ?}}|%\( ?ext ?\)[sS])$/
end end
@doc false
def json_exluded_fields do
~w(__meta__ __struct__ sources)a
end
defimpl Jason.Encoder, for: MediaProfile do
def encode(value, opts) do
value
|> Map.drop(MediaProfile.json_exluded_fields())
|> Jason.Encode.map(opts)
end
end
end end
@@ -15,7 +15,7 @@ defmodule Pinchflat.SlowIndexing.MediaCollectionIndexingWorker do
alias Pinchflat.Sources.Source alias Pinchflat.Sources.Source
alias Pinchflat.FastIndexing.FastIndexingWorker alias Pinchflat.FastIndexing.FastIndexingWorker
alias Pinchflat.SlowIndexing.SlowIndexingHelpers alias Pinchflat.SlowIndexing.SlowIndexingHelpers
alias Pinchflat.Notifications.SourceNotifications alias Pinchflat.Lifecycle.Notifications.SourceNotifications
@doc """ @doc """
Starts the source slow indexing worker and creates a task for the source. Starts the source slow indexing worker and creates a task for the source.
+16
View File
@@ -7,6 +7,8 @@ defmodule Pinchflat.Sources.Source do
import Ecto.Changeset import Ecto.Changeset
import Pinchflat.Utils.ChangesetUtils import Pinchflat.Utils.ChangesetUtils
alias __MODULE__
alias Pinchflat.Repo
alias Pinchflat.Tasks.Task alias Pinchflat.Tasks.Task
alias Pinchflat.Media.MediaItem alias Pinchflat.Media.MediaItem
alias Pinchflat.Profiles.MediaProfile alias Pinchflat.Profiles.MediaProfile
@@ -133,4 +135,18 @@ defmodule Pinchflat.Sources.Source do
def filepath_attributes do def filepath_attributes do
~w(nfo_filepath fanart_filepath poster_filepath banner_filepath)a ~w(nfo_filepath fanart_filepath poster_filepath banner_filepath)a
end end
@doc false
def json_exluded_fields do
~w(__meta__ __struct__ metadata tasks media_items)a
end
defimpl Jason.Encoder, for: Source do
def encode(value, opts) do
value
|> Repo.preload(:media_profile)
|> Map.drop(Source.json_exluded_fields())
|> Jason.Encode.map(opts)
end
end
end end
+15 -4
View File
@@ -237,7 +237,7 @@ defmodule Pinchflat.Sources do
if run_post_commit_tasks do if run_post_commit_tasks do
maybe_handle_media_tasks(changeset, source) maybe_handle_media_tasks(changeset, source)
maybe_run_indexing_task(changeset, source) maybe_run_indexing_task(changeset, source)
run_metadata_storage_task(source) maybe_run_metadata_storage_task(changeset, source)
end end
{:ok, source} {:ok, source}
@@ -276,9 +276,20 @@ defmodule Pinchflat.Sources do
end end
end end
# This runs every time to pick up any changes to the metadata defp maybe_run_metadata_storage_task(changeset, source) do
defp run_metadata_storage_task(source) do case {changeset.data, changeset.changes} do
SourceMetadataStorageWorker.kickoff_with_task(source) # If the changeset is new (not persisted), fetch metadata no matter what
{%{__meta__: %{state: :built}}, _} ->
SourceMetadataStorageWorker.kickoff_with_task(source)
# If the record has been persisted, only fetch metadata if the
# original_url has changed
{_, %{original_url: _}} ->
SourceMetadataStorageWorker.kickoff_with_task(source)
_ ->
:ok
end
end end
defp maybe_update_slow_indexing_task(changeset, source) do defp maybe_update_slow_indexing_task(changeset, source) do
+12 -3
View File
@@ -3,6 +3,8 @@ defmodule Pinchflat.Utils.CliUtils do
Utility methods for working with CLI executables Utility methods for working with CLI executables
""" """
require Logger
alias Pinchflat.Utils.StringUtils alias Pinchflat.Utils.StringUtils
@doc """ @doc """
@@ -11,15 +13,22 @@ defmodule Pinchflat.Utils.CliUtils do
commands if the job runner is cancelled. commands if the job runner is cancelled.
Delegates to `System.cmd/3` and any options/output Delegates to `System.cmd/3` and any options/output
are passed through. are passed through. Custom options can be passed in.
Custom options:
- logging_arg_override: if set, the passed value will be logged in place of
the actual arguments passed to the command
Returns {binary(), integer()} Returns {binary(), integer()}
""" """
def wrap_cmd(command, args, opts \\ []) do def wrap_cmd(command, args, passthrough_opts \\ [], opts \\ []) do
wrapper_command = Path.join(:code.priv_dir(:pinchflat), "cmd_wrapper.sh") wrapper_command = Path.join(:code.priv_dir(:pinchflat), "cmd_wrapper.sh")
actual_command = [command] ++ args actual_command = [command] ++ args
logging_arg_override = Keyword.get(opts, :logging_arg_override, Enum.join(args, " "))
System.cmd(wrapper_command, actual_command, opts) Logger.info("[command_wrapper]: #{command} called with: #{logging_arg_override}")
System.cmd(wrapper_command, actual_command, passthrough_opts)
end end
@doc """ @doc """
+15
View File
@@ -5,6 +5,21 @@ defmodule Pinchflat.Utils.FilesystemUtils do
alias Pinchflat.Media alias Pinchflat.Media
alias Pinchflat.Utils.StringUtils alias Pinchflat.Utils.StringUtils
@doc """
Checks if a file exists and has non-whitespace contents.
Returns boolean()
"""
def exists_and_nonempty?(filepath) do
case File.read(filepath) do
{:ok, contents} ->
String.trim(contents) != ""
_ ->
false
end
end
@doc """ @doc """
Generates a temporary file and returns its path. The file is empty and has the given type. Generates a temporary file and returns its path. The file is empty and has the given type.
Generates all the directories in the path if they don't exist. Generates all the directories in the path if they don't exist.
+16
View File
@@ -0,0 +1,16 @@
defmodule Pinchflat.Utils.NumberUtils do
@moduledoc """
Utility methods for working with numbers
"""
@doc """
Clamps a number between a minimum and maximum value
Returns integer() | float()
"""
def clamp(num, minimum, maximum) do
num
|> max(minimum)
|> min(maximum)
end
end
+12 -18
View File
@@ -3,8 +3,6 @@ defmodule Pinchflat.YtDlp.CommandRunner do
Runs yt-dlp commands using the `System.cmd/3` function Runs yt-dlp commands using the `System.cmd/3` function
""" """
require Logger
alias Pinchflat.Utils.CliUtils alias Pinchflat.Utils.CliUtils
alias Pinchflat.YtDlp.YtDlpCommandRunner alias Pinchflat.YtDlp.YtDlpCommandRunner
alias Pinchflat.Utils.FilesystemUtils, as: FSUtils alias Pinchflat.Utils.FilesystemUtils, as: FSUtils
@@ -30,12 +28,10 @@ defmodule Pinchflat.YtDlp.CommandRunner do
output_filepath = generate_output_filepath(addl_opts) output_filepath = generate_output_filepath(addl_opts)
print_to_file_opts = [{:print_to_file, output_template}, output_filepath] print_to_file_opts = [{:print_to_file, output_template}, output_filepath]
external_file_opts = build_external_file_options() user_configured_opts = cookie_file_options() ++ global_options()
# These must stay in exactly this order, hence why I'm giving it its own variable. # These must stay in exactly this order, hence why I'm giving it its own variable.
all_opts = command_opts ++ print_to_file_opts ++ external_file_opts all_opts = command_opts ++ print_to_file_opts ++ user_configured_opts
formatted_command_opts = [url] ++ CliUtils.parse_options(all_opts) formatted_command_opts = [url] ++ CliUtils.parse_options(all_opts)
Logger.info("[yt-dlp] called with: #{Enum.join(formatted_command_opts, " ")}")
case CliUtils.wrap_cmd(command, formatted_command_opts, stderr_to_stdout: true) do case CliUtils.wrap_cmd(command, formatted_command_opts, stderr_to_stdout: true) do
{_, 0} -> {_, 0} ->
@@ -58,7 +54,7 @@ defmodule Pinchflat.YtDlp.CommandRunner do
def version do def version do
command = backend_executable() command = backend_executable()
case System.cmd(command, ["--version"]) do case CliUtils.wrap_cmd(command, ["--version"]) do
{output, 0} -> {output, 0} ->
{:ok, String.trim(output)} {:ok, String.trim(output)}
@@ -74,23 +70,21 @@ defmodule Pinchflat.YtDlp.CommandRunner do
end end
end end
defp build_external_file_options do defp global_options do
[:windows_filenames]
end
defp cookie_file_options do
base_dir = Application.get_env(:pinchflat, :extras_directory) base_dir = Application.get_env(:pinchflat, :extras_directory)
filename_options_map = %{cookies: "cookies.txt"} filename_options_map = %{cookies: "cookies.txt"}
Enum.reduce(filename_options_map, [], fn {opt_name, filename}, acc -> Enum.reduce(filename_options_map, [], fn {opt_name, filename}, acc ->
filepath = Path.join(base_dir, filename) filepath = Path.join(base_dir, filename)
case File.read(filepath) do if FSUtils.exists_and_nonempty?(filepath) do
{:ok, file_data} -> [{opt_name, filepath} | acc]
if String.trim(file_data) != "" do else
[{opt_name, filepath} | acc] acc
else
acc
end
{:error, _} ->
acc
end end
end) end)
end end
+3 -3
View File
@@ -63,8 +63,8 @@ defmodule Pinchflat.YtDlp.Media do
|> response_to_struct() |> response_to_struct()
|> FunctionUtils.wrap_ok() |> FunctionUtils.wrap_ok()
res -> err ->
res err
end end
end end
@@ -87,7 +87,7 @@ defmodule Pinchflat.YtDlp.Media do
title: response["title"], title: response["title"],
description: response["description"], description: response["description"],
original_url: response["webpage_url"], original_url: response["webpage_url"],
livestream: response["was_live"], livestream: !!response["was_live"],
duration_seconds: response["duration"] && round(response["duration"]), duration_seconds: response["duration"] && round(response["duration"]),
short_form_content: response["webpage_url"] && short_form_content?(response), short_form_content: response["webpage_url"] && short_form_content?(response),
upload_date: response["upload_date"] && MetadataFileHelpers.parse_upload_date(response["upload_date"]) upload_date: response["upload_date"] && MetadataFileHelpers.parse_upload_date(response["upload_date"])
+10 -3
View File
@@ -50,8 +50,8 @@ defmodule Pinchflat.YtDlp.MediaCollection do
{:ok, Enum.filter(parsed_lines, &(&1 != nil))} {:ok, Enum.filter(parsed_lines, &(&1 != nil))}
res -> err ->
res err
end end
end end
@@ -68,7 +68,14 @@ defmodule Pinchflat.YtDlp.MediaCollection do
# `ignore_no_formats_error` is necessary because yt-dlp will error out if # `ignore_no_formats_error` is necessary because yt-dlp will error out if
# the first video has not released yet (ie: is a premier). We don't care about # the first video has not released yet (ie: is a premier). We don't care about
# available formats since we're just getting the source details # available formats since we're just getting the source details
command_opts = [:simulate, :skip_download, :ignore_no_formats_error, playlist_end: 1] ++ addl_opts default_opts = [
:simulate,
:skip_download,
:ignore_no_formats_error,
playlist_end: 1
]
command_opts = default_opts ++ addl_opts
output_template = "%(.{channel,channel_id,playlist_id,playlist_title,filename})j" output_template = "%(.{channel,channel_id,playlist_id,playlist_title,filename})j"
with {:ok, output} <- backend_runner().run(source_url, command_opts, output_template), with {:ok, output} <- backend_runner().run(source_url, command_opts, output_template),
+22 -17
View File
@@ -19,7 +19,6 @@ defmodule PinchflatWeb.CoreComponents do
import PinchflatWeb.Gettext import PinchflatWeb.Gettext
alias Phoenix.LiveView.JS alias Phoenix.LiveView.JS
alias PinchflatWeb.CustomComponents.TextComponents
@doc """ @doc """
Renders a modal. Renders a modal.
@@ -267,6 +266,7 @@ defmodule PinchflatWeb.CoreComponents do
multiple pattern placeholder readonly required rows size step) multiple pattern placeholder readonly required rows size step)
slot :inner_block slot :inner_block
slot :input_append
def input(%{field: %Phoenix.HTML.FormField{} = field} = assigns) do def input(%{field: %Phoenix.HTML.FormField{} = field} = assigns) do
assigns assigns
@@ -425,20 +425,23 @@ defmodule PinchflatWeb.CoreComponents do
<.label for={@id}> <.label for={@id}>
<%= @label %><span :if={@label_suffix} class="text-xs text-bodydark"><%= @label_suffix %></span> <%= @label %><span :if={@label_suffix} class="text-xs text-bodydark"><%= @label_suffix %></span>
</.label> </.label>
<input <div class="flex items-center">
type={@type} <input
name={@name} type={@type}
id={@id} name={@name}
value={Phoenix.HTML.Form.normalize_value(@type, @value)} id={@id}
class={[ value={Phoenix.HTML.Form.normalize_value(@type, @value)}
"w-full rounded-lg border-[1.5px] border-stroke bg-transparent px-5 py-3 font-normal text-black", class={[
"outline-none transition focus:border-primary active:border-primary disabled:cursor-default disabled:bg-whiter", "w-full rounded-lg border-[1.5px] px-5 py-3 font-normal border-form-strokedark bg-form-input",
"dark:border-form-strokedark dark:bg-form-input dark:text-white dark:focus:border-primary", "outline-none transition active:border-primary disabled:cursor-default disabled:bg-whiter",
@inputclass, "text-white focus:border-primary",
@errors != [] && "border-rose-400 focus:border-rose-400" @inputclass,
]} @errors != [] && "border-rose-400 focus:border-rose-400"
{@rest} ]}
/> {@rest}
/>
<%= render_slot(@input_append) %>
</div>
<.help :if={@help}><%= if @html_help, do: Phoenix.HTML.raw(@help), else: @help %></.help> <.help :if={@help}><%= if @html_help, do: Phoenix.HTML.raw(@help), else: @help %></.help>
<.error :for={msg <- @errors}><%= msg %></.error> <.error :for={msg <- @errors}><%= msg %></.error>
</div> </div>
@@ -638,9 +641,11 @@ defmodule PinchflatWeb.CoreComponents do
~H""" ~H"""
<ul> <ul>
<li :for={{k, v} <- @iterable_attributes} class="mb-2"> <li :for={{k, v} <- @iterable_attributes} class="mb-2 w-2/3">
<strong><%= k %>:</strong> <strong><%= k %>:</strong>
<TextComponents.inline_code><%= v %></TextComponents.inline_code> <code class="inline-block text-sm font-mono text-gray p-0.5 mx-0.5">
<%= v %>
</code>
</li> </li>
</ul> </ul>
""" """
@@ -89,4 +89,46 @@ defmodule PinchflatWeb.CustomComponents.ButtonComponents do
</div> </div>
""" """
end end
@doc """
Render a button with an icon. Optionally include a tooltip.
## Examples
<.icon_button icon_name="hero-check" tooltip="Complete" />
"""
attr :icon_name, :string, required: true
attr :class, :string, default: ""
attr :tooltip, :string, default: nil
attr :rest, :global
def icon_button(assigns) do
~H"""
<div class="group relative inline-block">
<button
class={[
"flex justify-center items-center rounded-lg ",
"bg-form-input border-2 border-strokedark",
"hover:bg-meta-4 hover:border-form-strokedark",
@class
]}
type="button"
{@rest}
>
<CoreComponents.icon name={@icon_name} class="text-stroke" />
</button>
<div
:if={@tooltip}
class={[
"hidden absolute left-1/2 top-full z-20 mt-3 -translate-x-1/2 whitespace-nowrap rounded-md",
"px-4.5 py-1.5 text-sm font-medium opacity-0 drop-shadow-4 group-hover:opacity-100 group-hover:block bg-meta-4"
]}
>
<span class="border-light absolute -top-1 left-1/2 -z-10 h-2 w-2 -translate-x-1/2 rotate-45 rounded-sm bg-meta-4">
</span>
<span><%= @tooltip %></span>
</div>
</div>
"""
end
end end
@@ -2,6 +2,8 @@ defmodule PinchflatWeb.CustomComponents.TableComponents do
@moduledoc false @moduledoc false
use Phoenix.Component use Phoenix.Component
alias PinchflatWeb.CoreComponents
@doc """ @doc """
Renders a table component with the given rows and columns. Renders a table component with the given rows and columns.
@@ -50,4 +52,52 @@ defmodule PinchflatWeb.CustomComponents.TableComponents do
</table> </table>
""" """
end end
@doc """
Renders simple pagination controls for a table in a liveview.
## Examples
<.live_pagination_controls page_number={@page} total_pages={@total_pages} />
"""
attr :page_number, :integer, default: 1
attr :total_pages, :integer, default: 1
def live_pagination_controls(assigns) do
~H"""
<nav>
<ul class="flex flex-wrap items-center">
<li>
<span
class={[
"flex h-8 w-8 items-center justify-center rounded",
@page_number != 1 && "cursor-pointer hover:bg-primary hover:text-white",
@page_number == 1 && "cursor-not-allowed"
]}
phx-click={@page_number != 1 && "page_change"}
phx-value-direction="dec"
>
<CoreComponents.icon name="hero-chevron-left" />
</span>
</li>
<li>
<span class="mx-2">Page <%= @page_number %> of <%= @total_pages %></span>
</li>
<li>
<span
class={[
"flex h-8 w-8 items-center justify-center rounded",
@page_number != @total_pages && "cursor-pointer hover:bg-primary hover:text-white",
@page_number == @total_pages && "cursor-not-allowed"
]}
phx-click={@page_number != @total_pages && "page_change"}
phx-value-direction="inc"
>
<CoreComponents.icon name="hero-chevron-right" />
</span>
</li>
</ul>
</nav>
"""
end
end end
@@ -35,11 +35,12 @@ defmodule PinchflatWeb.CustomComponents.TextComponents do
Renders a subtle link with the given href and content. Renders a subtle link with the given href and content.
""" """
attr :href, :string, required: true attr :href, :string, required: true
attr :target, :string, default: "_self"
slot :inner_block slot :inner_block
def subtle_link(assigns) do def subtle_link(assigns) do
~H""" ~H"""
<.link href={@href} class="underline decoration-bodydark decoration-1 hover:decoration-white"> <.link href={@href} target={@target} class="underline decoration-bodydark decoration-1 hover:decoration-white">
<%= render_slot(@inner_block) %> <%= render_slot(@inner_block) %>
</.link> </.link>
""" """
@@ -59,4 +60,22 @@ defmodule PinchflatWeb.CustomComponents.TextComponents do
</.link> </.link>
""" """
end end
@doc """
Renders a block of text with each line broken into a separate span.
"""
attr :text, :string, required: true
def break_on_newline(assigns) do
broken_text =
assigns.text
|> String.split("\n", trim: false)
|> Enum.intersperse(Phoenix.HTML.Tag.tag(:span, class: "inline-block mt-2"))
assigns = Map.put(assigns, :text, broken_text)
~H"""
<span><%= @text %></span>
"""
end
end end
@@ -54,10 +54,22 @@
</span> </span>
</li> </li>
<li> <li>
<span class="group relative flex items-center gap-2.5 px-4 pt-2 text-sm"> <span
Pinchflat v<%= Application.spec(:pinchflat)[:vsn] %> class="group relative flex items-center gap-2.5 px-4 pt-2 text-sm"
x-on:click={"markVersionAsSeen('#{Application.spec(:pinchflat)[:vsn]}')"}
>
<span>Pinchflat v<%= Application.spec(:pinchflat)[:vsn] %></span>
<a
href="https://github.com/kieraneglin/pinchflat/releases"
target="_blank"
class="bg-meta-2 text-boxdark px-1.5 rounded-full text-xs"
x-cloak
x-show={"!isVersionSeen('#{Application.spec(:pinchflat)[:vsn]}')"}
>
NEW
</a>
</span> </span>
<span class="group relative flex items-center gap-2.5 px-4 text-sm"> <span class="group relative flex items-center gap-2.5 px-4 pt-2 text-sm">
yt-dlp <%= Settings.get!(:yt_dlp_version) %> yt-dlp <%= Settings.get!(:yt_dlp_version) %>
</span> </span>
</li> </li>
@@ -1,4 +1,19 @@
<.button_dropdown text="Actions" class="justify-center w-full sm:w-50"> <.button_dropdown text="Actions" class="justify-center w-full sm:w-50">
<:option>
<span x-data="{ copied: false }" x-on:click={~s"
copyWithCallbacks(
`#{Jason.Formatter.pretty_print(Phoenix.json_library().encode!(@media_item))}`,
() => copied = true,
() => copied = false
)
"}>
Copy JSON
<span x-show="copied" x-transition.duration.150ms><.icon name="hero-check" class="ml-2 h-4 w-4" /></span>
</span>
</:option>
<:option>
<div class="h-px w-full bg-bodydark2"></div>
</:option>
<:option> <:option>
<.link <.link
href={~p"/sources/#{@media_item.source_id}/media/#{@media_item}/force_download"} href={~p"/sources/#{@media_item.source_id}/media/#{@media_item}/force_download"}
@@ -5,7 +5,7 @@
</div> </div>
<div class="rounded-sm border border-stroke bg-white px-5 pb-2.5 pt-6 shadow-default dark:border-strokedark dark:bg-boxdark sm:px-7.5 xl:pb-1"> <div class="rounded-sm border border-stroke bg-white px-5 pb-2.5 pt-6 shadow-default dark:border-strokedark dark:bg-boxdark sm:px-7.5 xl:pb-1">
<div class="max-w-full overflow-x-auto"> <div class="max-w-full">
<div class="flex flex-col gap-10"> <div class="flex flex-col gap-10">
<.media_item_form changeset={@changeset} action={~p"/sources/#{@media_item.source_id}/media/#{@media_item}"} /> <.media_item_form changeset={@changeset} action={~p"/sources/#{@media_item.source_id}/media/#{@media_item}"} />
</div> </div>
@@ -1,12 +1,12 @@
<%= if media_type(@media_item) == :video do %> <%= if media_type(@media_item) == :video do %>
<video controls class="max-h-96 w-full lg:w-2/3 xl:w-1/2"> <video controls class="max-h-96 w-full">
<source src={~p"/media/#{@media_item.uuid}/stream"} type="video/mp4" /> <source src={~p"/media/#{@media_item.uuid}/stream"} type="video/mp4" />
Your browser does not support the video element. Your browser does not support the video element.
</video> </video>
<% end %> <% end %>
<%= if media_type(@media_item) == :audio do %> <%= if media_type(@media_item) == :audio do %>
<audio controls class="w-full lg:w-2/3 xl:w-1/2"> <audio controls class="w-full">
<source src={~p"/media/#{@media_item.uuid}/stream"} type="audio/mpeg" /> <source src={~p"/media/#{@media_item.uuid}/stream"} type="audio/mpeg" />
Your browser does not support the audio element. Your browser does not support the audio element.
</audio> </audio>
@@ -17,29 +17,39 @@
</nav> </nav>
</div> </div>
<div class="rounded-sm border border-stroke bg-white py-5 pt-6 shadow-default dark:border-strokedark dark:bg-boxdark px-7.5"> <div class="rounded-sm border border-stroke bg-white py-5 pt-6 shadow-default dark:border-strokedark dark:bg-boxdark px-7.5">
<div class="max-w-full overflow-x-auto"> <div class="max-w-full">
<.tabbed_layout> <.tabbed_layout>
<:tab_append> <:tab_append>
<.actions_dropdown media_item={@media_item} /> <.actions_dropdown media_item={@media_item} />
</:tab_append> </:tab_append>
<:tab title="Attributes"> <:tab title="Media">
<div class="flex flex-col gap-10 dark:text-white"> <div class="flex flex-col gap-10 dark:text-white">
<%= if media_file_exists?(@media_item) do %> <%= if media_file_exists?(@media_item) do %>
<h3 class="font-bold text-xl">Preview</h3> <section class="grid grid-cols-1 xl:grid-cols-2 xl:gap-6 mt-6">
<.media_preview media_item={@media_item} /> <div>
<.media_preview media_item={@media_item} />
</div>
<aside class="mt-4 xl:mt-0">
<div>Uploaded: <%= @media_item.upload_date %></div>
<div :if={URI.parse(@media_item.original_url).scheme =~ "http"}>
<.subtle_link href={@media_item.original_url} target="_blank">Open Original</.subtle_link>
</div>
<div class="mt-4 text-bodydark">
<.break_on_newline text={@media_item.description} />
</div>
</aside>
</section>
<% end %> <% end %>
<h2 class="font-bold text-2xl"><%= @media_item.title %></h2> <h3 class="font-bold text-xl mt-6">Raw Attributes</h3>
<h3 class="font-bold text-xl">Attributes</h3>
<section> <section>
<strong>Source:</strong> <strong>Source:</strong>
<.subtle_link href={~p"/sources/#{@media_item.source_id}"}> <.subtle_link href={~p"/sources/#{@media_item.source_id}"}>
<%= @media_item.source.custom_name %> <%= @media_item.source.custom_name %>
</.subtle_link> </.subtle_link>
<.list_items_from_map map={Map.from_struct(@media_item)} />
</section> </section>
<.list_items_from_map map={Map.from_struct(@media_item)} />
</div> </div>
</:tab> </:tab>
<:tab title="Tasks"> <:tab title="Tasks">
@@ -63,7 +63,11 @@ defmodule PinchflatWeb.MediaProfiles.MediaProfileHTML do
source_collection_name: source_collection_name:
"the YouTube name of the sources that use this profile (often the same as source_custom_name)", "the YouTube name of the sources that use this profile (often the same as source_custom_name)",
source_collection_type: "the collection type of the sources using this profile. Either 'channel' or 'playlist'", source_collection_type: "the collection type of the sources using this profile. Either 'channel' or 'playlist'",
artist_name: "the name of the artist with fallbacks to other uploader fields" artist_name: "the name of the artist with fallbacks to other uploader fields",
season_from_date: "alias for upload_year",
season_episode_from_date: "the upload date formatted as sYYYYeMMDD",
season_episode_index_from_date:
"the upload date formatted as sYYYYeMMDDII where II is an index to prevent date collisions"
} }
end end
@@ -94,7 +98,7 @@ defmodule PinchflatWeb.MediaProfiles.MediaProfileHTML do
end end
defp media_center_output_template do defp media_center_output_template do
"/shows/{{ source_custom_name }}/Season {{ season_from_date }}/{{ season_episode_from_date }} - {{ title }}.{{ ext }}" "/shows/{{ source_custom_name }}/Season {{ season_from_date }}/{{ season_episode_index_from_date }} - {{ title }}.{{ ext }}"
end end
defp audio_output_template do defp audio_output_template do
@@ -0,0 +1,36 @@
<.button_dropdown text="Actions" class="justify-center w-full sm:w-50">
<:option>
<span x-data="{ copied: false }" x-on:click={~s"
copyWithCallbacks(
`#{Jason.Formatter.pretty_print(Phoenix.json_library().encode!(@media_profile))}`,
() => copied = true,
() => copied = false
)
"}>
Copy JSON
<span x-show="copied" x-transition.duration.150ms><.icon name="hero-check" class="ml-2 h-4 w-4" /></span>
</span>
</:option>
<:option>
<div class="h-px w-full bg-bodydark2"></div>
</:option>
<:option>
<.link
href={~p"/media_profiles/#{@media_profile}"}
method="delete"
data-confirm="Are you sure you want to delete this profile and all its sources (leaving files in place)? This cannot be undone."
>
Delete Profile
</.link>
</:option>
<:option>
<.link
href={~p"/media_profiles/#{@media_profile}?delete_files=true"}
method="delete"
data-confirm="Are you sure you want to delete this profile, all its sources, and its files on disk? This cannot be undone."
class="mt-5 md:mt-0"
>
Delete Profile + Files
</.link>
</:option>
</.button_dropdown>
@@ -8,7 +8,7 @@
</div> </div>
<div class="rounded-sm border border-stroke bg-white px-5 pb-2.5 pt-6 shadow-default dark:border-strokedark dark:bg-boxdark sm:px-7.5 xl:pb-1"> <div class="rounded-sm border border-stroke bg-white px-5 pb-2.5 pt-6 shadow-default dark:border-strokedark dark:bg-boxdark sm:px-7.5 xl:pb-1">
<div class="max-w-full overflow-x-auto"> <div class="max-w-full">
<div class="flex flex-col gap-10"> <div class="flex flex-col gap-10">
<.media_profile_form changeset={@changeset} action={~p"/media_profiles/#{@media_profile}"} /> <.media_profile_form changeset={@changeset} action={~p"/media_profiles/#{@media_profile}"} />
</div> </div>
@@ -6,7 +6,7 @@
</div> </div>
<div class="rounded-sm border border-stroke bg-white px-5 pb-2.5 pt-6 shadow-default dark:border-strokedark dark:bg-boxdark sm:px-7.5 xl:pb-1"> <div class="rounded-sm border border-stroke bg-white px-5 pb-2.5 pt-6 shadow-default dark:border-strokedark dark:bg-boxdark sm:px-7.5 xl:pb-1">
<div class="max-w-full overflow-x-auto"> <div class="max-w-full">
<div class="flex flex-col gap-10"> <div class="flex flex-col gap-10">
<.media_profile_form changeset={@changeset} action={~p"/media_profiles"} /> <.media_profile_form changeset={@changeset} action={~p"/media_profiles"} />
</div> </div>
@@ -17,35 +17,15 @@
</nav> </nav>
</div> </div>
<div class="rounded-sm border border-stroke bg-white px-5 pb-2.5 pt-6 shadow-default dark:border-strokedark dark:bg-boxdark sm:px-7.5 xl:pb-1"> <div class="rounded-sm border border-stroke bg-white px-5 pb-2.5 pt-6 shadow-default dark:border-strokedark dark:bg-boxdark sm:px-7.5 xl:pb-1">
<div class="max-w-full overflow-x-auto"> <div class="max-w-full">
<.tabbed_layout> <.tabbed_layout>
<:tab_append> <:tab_append>
<.button_dropdown text="Actions" class="justify-center w-full sm:w-50"> <.actions_dropdown media_profile={@media_profile} />
<:option>
<.link
href={~p"/media_profiles/#{@media_profile}"}
method="delete"
data-confirm="Are you sure you want to delete this profile and all its sources (leaving files in place)? This cannot be undone."
>
Delete Profile
</.link>
</:option>
<:option>
<.link
href={~p"/media_profiles/#{@media_profile}?delete_files=true"}
method="delete"
data-confirm="Are you sure you want to delete this profile, all its sources, and its files on disk? This cannot be undone."
class="mt-5 md:mt-0"
>
Delete Profile + Files
</.link>
</:option>
</.button_dropdown>
</:tab_append> </:tab_append>
<:tab title="Attributes"> <:tab title="Media Profile">
<div class="flex flex-col gap-10 text-white"> <div class="flex flex-col gap-10 text-white">
<h3 class="font-bold text-xl">Attributes for "<%= @media_profile.name %>"</h3> <h3 class="font-bold text-xl mt-6">Raw Attributes</h3>
<.list_items_from_map map={Map.from_struct(@media_profile)} /> <.list_items_from_map map={Map.from_struct(@media_profile)} />
</div> </div>
</:tab> </:tab>
@@ -16,12 +16,12 @@
</a> </a>
</div> </div>
<div class="rounded-sm border px-7.5 py-6 shadow-default border-strokedark bg-boxdark"> <div class="rounded-sm border px-7.5 py-6 shadow-default border-strokedark bg-boxdark">
<a href="#" class="mt-4 flex flex-col items-center justify-center"> <span class="mt-4 flex flex-col items-center justify-center">
<span class="text-md font-medium">Media Item(s)</span> <span class="text-md font-medium">Media Item(s)</span>
<h4 class="text-title-md font-bold text-white"> <h4 class="text-title-md font-bold text-white">
<%= @media_item_count %> <%= @media_item_count %>
</h4> </h4>
</a> </span>
</div> </div>
<span class="text-strokedark">I know this page isn't super useful yet, but give it time :&rpar;</span> <span class="text-strokedark">I know this page isn't super useful yet, but give it time :&rpar;</span>
</div> </div>
@@ -2,8 +2,9 @@ defmodule PinchflatWeb.Podcasts.PodcastController do
use PinchflatWeb, :controller use PinchflatWeb, :controller
alias Pinchflat.Repo alias Pinchflat.Repo
alias Pinchflat.Media.MediaQuery
alias Pinchflat.Sources.Source alias Pinchflat.Sources.Source
alias Pinchflat.Media.MediaItem
alias Pinchflat.Media.MediaQuery
alias Pinchflat.Podcasts.RssFeedBuilder alias Pinchflat.Podcasts.RssFeedBuilder
alias Pinchflat.Podcasts.PodcastHelpers alias Pinchflat.Podcasts.PodcastHelpers
@@ -40,4 +41,16 @@ defmodule PinchflatWeb.Podcasts.PodcastController do
|> send_file(200, filepath) |> send_file(200, filepath)
end end
end end
def episode_image(conn, %{"uuid" => uuid}) do
media_item = Repo.get_by!(MediaItem, uuid: uuid)
if media_item.thumbnail_filepath && File.exists?(media_item.thumbnail_filepath) do
conn
|> put_resp_content_type(MIME.from_path(media_item.thumbnail_filepath))
|> send_file(200, media_item.thumbnail_filepath)
else
send_resp(conn, 404, "Image not found")
end
end
end end
@@ -5,7 +5,7 @@
</div> </div>
<div class="rounded-sm border border-stroke bg-white px-5 py-5 shadow-default dark:border-strokedark dark:bg-boxdark sm:px-7.5"> <div class="rounded-sm border border-stroke bg-white px-5 py-5 shadow-default dark:border-strokedark dark:bg-boxdark sm:px-7.5">
<div class="max-w-full overflow-x-auto"> <div class="max-w-full">
<div class="flex flex-col gap-10 dark:text-white"> <div class="flex flex-col gap-10 dark:text-white">
<%= if match?([_|_], @search_results) do %> <%= if match?([_|_], @search_results) do %>
<.table rows={@search_results} table_class="text-black dark:text-white"> <.table rows={@search_results} table_class="text-black dark:text-white">
@@ -6,6 +6,7 @@ defmodule PinchflatWeb.Settings.SettingHTML do
@doc """ @doc """
Renders a setting form. Renders a setting form.
""" """
attr :conn, Plug.Conn, required: true
attr :changeset, Ecto.Changeset, required: true attr :changeset, Ecto.Changeset, required: true
attr :action, :string, required: true attr :action, :string, required: true
@@ -0,0 +1,55 @@
defmodule Pinchflat.Settings.AppriseServerLive do
use PinchflatWeb, :live_view
alias PinchflatWeb.Settings.SettingHTML
def render(assigns) do
~H"""
<.input
type="text"
id="setting_apprise_server"
name="setting[apprise_server]"
value={@value}
label="Apprise Server"
help={SettingHTML.apprise_server_help()}
html_help={true}
inputclass="font-mono text-sm mr-4"
placeholder="https://discordapp.com/api/webhooks/{WebhookID}/{WebhookToken}"
phx-change="apprise_server_changed"
>
<:input_append>
<.icon_button icon_name={@icon_name} class="h-12 w-12" phx-click="send_apprise_test" tooltip={@tooltip} />
</:input_append>
</.input>
"""
end
def mount(_params, session, socket) do
new_assigns = %{
value: session["value"],
icon_name: "hero-paper-airplane",
tooltip: "Send Test"
}
{:ok, assign(socket, new_assigns)}
end
def handle_event("send_apprise_test", _params, %{assigns: assigns} = socket) do
backend_runner().run([assigns.value], title: "Pinchflat Test", body: "This is a test message from Pinchflat")
Process.send_after(self(), :reset_button_icon, 4_000)
{:noreply, assign(socket, %{icon_name: "hero-check", tooltip: "Sent!"})}
end
def handle_event("apprise_server_changed", %{"setting" => setting}, socket) do
{:noreply, assign(socket, %{value: setting["apprise_server"]})}
end
def handle_info(:reset_button_icon, socket) do
{:noreply, assign(socket, %{icon_name: "hero-paper-airplane", tooltip: "Send Test"})}
end
defp backend_runner do
Application.get_env(:pinchflat, :apprise_runner)
end
end
@@ -3,19 +3,17 @@
Oops, something went wrong! Please check the errors below. Oops, something went wrong! Please check the errors below.
</.error> </.error>
<h3 class="mt-8 text-2xl text-black dark:text-white"> <h3 class="mt-2 md:mt-8 text-2xl text-black dark:text-white">
Notification Settings Notification Settings
</h3> </h3>
<.input <section>
field={f[:apprise_server]} <%= live_render(
type="text" @conn,
label="Apprise Server" Pinchflat.Settings.AppriseServerLive,
help={apprise_server_help()} session: %{"value" => f[:apprise_server].value}
html_help={true} ) %>
inputclass="font-mono text-sm" </section>
placeholder="https://discordapp.com/api/webhooks/{WebhookID}/{WebhookToken}"
/>
<.button class="my-10 sm:mb-7.5 w-full sm:w-auto" rounding="rounded-lg">Save Settings</.button> <.button class="mt-10 mb-4 sm:mb-8 w-full sm:w-auto" rounding="rounded-lg">Save Settings</.button>
</.simple_form> </.simple_form>
@@ -6,7 +6,7 @@
</div> </div>
</div> </div>
<div class="rounded-sm border border-stroke bg-white px-5 py-5 shadow-default dark:border-strokedark dark:bg-boxdark sm:px-7.5"> <div class="rounded-sm border border-stroke bg-white px-5 py-5 shadow-default dark:border-strokedark dark:bg-boxdark sm:px-7.5">
<div class="max-w-full overflow-x-auto"> <div class="max-w-full">
<.setting_form changeset={@changeset} action={~p"/settings"} /> <.setting_form conn={@conn} changeset={@changeset} action={~p"/settings"} />
</div> </div>
</div> </div>
@@ -6,12 +6,11 @@ defmodule PinchflatWeb.Sources.SourceController do
alias Pinchflat.Repo alias Pinchflat.Repo
alias Pinchflat.Tasks alias Pinchflat.Tasks
alias Pinchflat.Sources alias Pinchflat.Sources
alias Pinchflat.MediaQuery
alias Pinchflat.Sources.Source alias Pinchflat.Sources.Source
alias Pinchflat.Media.MediaQuery
alias Pinchflat.Profiles.MediaProfile alias Pinchflat.Profiles.MediaProfile
alias Pinchflat.Downloading.DownloadingHelpers alias Pinchflat.Downloading.DownloadingHelpers
alias Pinchflat.SlowIndexing.SlowIndexingHelpers alias Pinchflat.SlowIndexing.SlowIndexingHelpers
alias Pinchflat.Metadata.SourceMetadataStorageWorker
def index(conn, _params) do def index(conn, _params) do
sources = sources =
@@ -60,29 +59,7 @@ defmodule PinchflatWeb.Sources.SourceController do
|> Tasks.list_tasks_for(nil, [:executing, :available, :scheduled, :retryable]) |> Tasks.list_tasks_for(nil, [:executing, :available, :scheduled, :retryable])
|> Repo.preload(:job) |> Repo.preload(:job)
pending_media = render(conn, :show, source: source, pending_tasks: pending_tasks)
MediaQuery.new()
|> MediaQuery.for_source(source)
|> MediaQuery.where_pending_download()
|> order_by(desc: :id)
|> limit(100)
|> Repo.all()
downloaded_media =
MediaQuery.new()
|> MediaQuery.for_source(source)
|> MediaQuery.with_media_filepath()
|> order_by(desc: :id)
|> limit(100)
|> Repo.all()
render(conn, :show,
source: source,
pending_tasks: pending_tasks,
pending_media: pending_media,
downloaded_media: downloaded_media,
total_downloaded: total_downloaded_for(source)
)
end end
def edit(conn, %{"id" => id}) do def edit(conn, %{"id" => id}) do
@@ -128,20 +105,38 @@ defmodule PinchflatWeb.Sources.SourceController do
end end
def force_download(conn, %{"source_id" => id}) do def force_download(conn, %{"source_id" => id}) do
source = Sources.get_source!(id) wrap_forced_action(
DownloadingHelpers.enqueue_pending_download_tasks(source) conn,
id,
conn "Forcing download of pending media items.",
|> put_flash(:info, "Forced download of pending media items.") &DownloadingHelpers.enqueue_pending_download_tasks/1
|> redirect(to: ~p"/sources/#{source}") )
end end
def force_index(conn, %{"source_id" => id}) do def force_index(conn, %{"source_id" => id}) do
source = Sources.get_source!(id) wrap_forced_action(
SlowIndexingHelpers.kickoff_indexing_task(source, %{force: true}) conn,
id,
"Index enqueued.",
&SlowIndexingHelpers.kickoff_indexing_task(&1, %{force: true})
)
end
def force_metadata_refresh(conn, %{"source_id" => id}) do
wrap_forced_action(
conn,
id,
"Metadata refresh enqueued.",
&SourceMetadataStorageWorker.kickoff_with_task/1
)
end
defp wrap_forced_action(conn, source_id, message, fun) do
source = Sources.get_source!(source_id)
fun.(source)
conn conn
|> put_flash(:info, "Index enqueued.") |> put_flash(:info, message)
|> redirect(to: ~p"/sources/#{source}") |> redirect(to: ~p"/sources/#{source}")
end end
@@ -151,13 +146,6 @@ defmodule PinchflatWeb.Sources.SourceController do
|> Repo.all() |> Repo.all()
end end
defp total_downloaded_for(source) do
MediaQuery.new()
|> MediaQuery.for_source(source)
|> MediaQuery.with_media_filepath()
|> Repo.aggregate(:count, :id)
end
defp get_onboarding_layout do defp get_onboarding_layout do
if Settings.get!(:onboarding) do if Settings.get!(:onboarding) do
{Layouts, :onboarding} {Layouts, :onboarding}
@@ -36,6 +36,15 @@ defmodule PinchflatWeb.Sources.SourceHTML do
|> Phoenix.json_library().encode!() |> Phoenix.json_library().encode!()
end end
def title_filter_regex_help do
url = "https://github.com/nalgeon/sqlean/blob/main/docs/regexp.md#supported-syntax"
classes = "underline decoration-bodydark decoration-1 hover:decoration-white"
"""
A PCRE-compatible regex. Only media with titles that match this regex will be downloaded. <a href="#{url}" class="#{classes}" target="_blank">See here</a> for syntax
"""
end
def output_path_template_override_help do def output_path_template_override_help do
help_button_classes = "underline decoration-bodydark decoration-1 hover:decoration-white cursor-pointer" help_button_classes = "underline decoration-bodydark decoration-1 hover:decoration-white cursor-pointer"
help_button = ~s{<span class="#{help_button_classes}" x-on:click="$dispatch('load-template')">Click here</span>} help_button = ~s{<span class="#{help_button_classes}" x-on:click="$dispatch('load-template')">Click here</span>}
@@ -3,15 +3,32 @@
<span <span
x-data="{ copied: false }" x-data="{ copied: false }"
x-on:click={" x-on:click={"
window.copyTextToClipboard('#{rss_feed_url(@conn, @source)}') copyWithCallbacks(
copied = true '#{rss_feed_url(@conn, @source)}',
setTimeout(() => copied = false, 4000) () => copied = true,
"} () => copied = false
)
"}
> >
Copy RSS Feed Copy RSS Feed
<span x-show="copied" x-transition.duration.150ms><.icon name="hero-check" class="ml-2 h-4 w-4" /></span> <span x-show="copied" x-transition.duration.150ms><.icon name="hero-check" class="ml-2 h-4 w-4" /></span>
</span> </span>
</:option> </:option>
<:option>
<span x-data="{ copied: false }" x-on:click={~s"
copyWithCallbacks(
`#{Jason.Formatter.pretty_print(Phoenix.json_library().encode!(@source))}`,
() => copied = true,
() => copied = false
)
"}>
Copy JSON
<span x-show="copied" x-transition.duration.150ms><.icon name="hero-check" class="ml-2 h-4 w-4" /></span>
</span>
</:option>
<:option>
<div class="h-px w-full bg-bodydark2"></div>
</:option>
<:option :if={@source.download_media}> <:option :if={@source.download_media}>
<.link <.link
href={~p"/sources/#{@source}/force_download"} href={~p"/sources/#{@source}/force_download"}
@@ -30,6 +47,15 @@
Force Index Force Index
</.link> </.link>
</:option> </:option>
<:option>
<.link
href={~p"/sources/#{@source}/force_metadata_refresh"}
method="post"
data-confirm="Are you sure you want to refresh this source's metadata?"
>
Refresh Metadata
</.link>
</:option>
<:option> <:option>
<div class="h-px w-full bg-bodydark2"></div> <div class="h-px w-full bg-bodydark2"></div>
</:option> </:option>
@@ -8,7 +8,7 @@
</div> </div>
<div class="rounded-sm border border-stroke bg-white px-5 pb-2.5 pt-6 shadow-default dark:border-strokedark dark:bg-boxdark sm:px-7.5 xl:pb-1"> <div class="rounded-sm border border-stroke bg-white px-5 pb-2.5 pt-6 shadow-default dark:border-strokedark dark:bg-boxdark sm:px-7.5 xl:pb-1">
<div class="max-w-full overflow-x-auto"> <div class="max-w-full">
<div class="flex flex-col gap-10"> <div class="flex flex-col gap-10">
<.source_form changeset={@changeset} media_profiles={@media_profiles} action={~p"/sources/#{@source}"} /> <.source_form changeset={@changeset} media_profiles={@media_profiles} action={~p"/sources/#{@source}"} />
</div> </div>
@@ -0,0 +1,101 @@
defmodule Pinchflat.Sources.MediaItemTableLive do
use PinchflatWeb, :live_view
import Ecto.Query, warn: false
alias Pinchflat.Repo
alias Pinchflat.Sources
alias Pinchflat.Media.MediaQuery
alias Pinchflat.Utils.NumberUtils
@limit 10
def render(%{records: []} = assigns) do
~H"""
<div class="mb-4 flex items-center">
<.icon_button icon_name="hero-arrow-path" class="h-10 w-10" phx-click="reload_page" />
<p class="ml-2">Nothing Here!</p>
</div>
"""
end
def render(assigns) do
~H"""
<div>
<span class="mb-4 flex items-center">
<.icon_button icon_name="hero-arrow-path" class="h-10 w-10" phx-click="reload_page" tooltip="Refresh" />
<span class="ml-2">Showing <%= length(@records) %> of <%= @total_record_count %></span>
</span>
<.table rows={@records} table_class="text-white">
<:col :let={media_item} label="Title">
<.subtle_link href={~p"/sources/#{@source.id}/media/#{media_item.id}"}>
<%= StringUtils.truncate(media_item.title, 50) %>
</.subtle_link>
</:col>
<:col :let={media_item} label="" class="flex place-content-evenly">
<.icon_link href={~p"/sources/#{@source.id}/media/#{media_item.id}"} icon="hero-eye" class="mx-1" />
<.icon_link href={~p"/sources/#{@source.id}/media/#{media_item.id}/edit"} icon="hero-pencil-square" class="mx-1" />
</:col>
</.table>
<section class="flex justify-center mt-5">
<.live_pagination_controls page_number={@page} total_pages={@total_pages} />
</section>
</div>
"""
end
def mount(_params, session, socket) do
page = 1
media_state = session["media_state"]
source = Sources.get_source!(session["source_id"])
base_query = generate_base_query(source, media_state)
pagination_attrs = fetch_pagination_attributes(base_query, page)
{:ok, assign(socket, Map.merge(pagination_attrs, %{base_query: base_query, source: source}))}
end
def handle_event("page_change", %{"direction" => direction}, %{assigns: assigns} = socket) do
direction = if direction == "inc", do: 1, else: -1
new_page = assigns.page + direction
new_assigns = fetch_pagination_attributes(assigns.base_query, new_page)
{:noreply, assign(socket, new_assigns)}
end
def handle_event("reload_page", _params, %{assigns: assigns} = socket) do
new_assigns = fetch_pagination_attributes(assigns.base_query, assigns.page)
{:noreply, assign(socket, new_assigns)}
end
defp fetch_pagination_attributes(base_query, page) do
total_record_count = Repo.aggregate(base_query, :count, :id)
total_pages = max(ceil(total_record_count / @limit), 1)
page = NumberUtils.clamp(page, 1, total_pages)
records = fetch_records(base_query, page)
%{page: page, total_pages: total_pages, records: records, total_record_count: total_record_count}
end
defp fetch_records(base_query, page) do
offset = (page - 1) * @limit
base_query
|> limit(^@limit)
|> offset(^offset)
|> Repo.all()
end
defp generate_base_query(source, "pending") do
MediaQuery.new()
|> MediaQuery.for_source(source)
|> MediaQuery.where_pending_download()
|> order_by(desc: :id)
end
defp generate_base_query(source, "downloaded") do
MediaQuery.new()
|> MediaQuery.for_source(source)
|> MediaQuery.with_media_filepath()
|> order_by(desc: :id)
end
end
@@ -6,7 +6,7 @@
</div> </div>
<div class="rounded-sm border border-stroke bg-white px-5 pb-2.5 pt-6 shadow-default dark:border-strokedark dark:bg-boxdark sm:px-7.5 xl:pb-1"> <div class="rounded-sm border border-stroke bg-white px-5 pb-2.5 pt-6 shadow-default dark:border-strokedark dark:bg-boxdark sm:px-7.5 xl:pb-1">
<div class="max-w-full overflow-x-auto"> <div class="max-w-full">
<div class="flex flex-col gap-10"> <div class="flex flex-col gap-10">
<.source_form changeset={@changeset} media_profiles={@media_profiles} action={~p"/sources"} /> <.source_form changeset={@changeset} media_profiles={@media_profiles} action={~p"/sources"} />
</div> </div>
@@ -17,15 +17,15 @@
</nav> </nav>
</div> </div>
<div class="rounded-sm border border-stroke bg-white px-5 py-5 shadow-default dark:border-strokedark dark:bg-boxdark sm:px-7.5"> <div class="rounded-sm border border-stroke bg-white px-5 py-5 shadow-default dark:border-strokedark dark:bg-boxdark sm:px-7.5">
<div class="max-w-full overflow-x-auto"> <div class="max-w-full">
<.tabbed_layout> <.tabbed_layout>
<:tab_append> <:tab_append>
<.actions_dropdown source={@source} conn={@conn} /> <.actions_dropdown source={@source} conn={@conn} />
</:tab_append> </:tab_append>
<:tab title="Attributes"> <:tab title="Source">
<div class="flex flex-col gap-10 text-white"> <div class="flex flex-col gap-10 text-white">
<h3 class="font-bold text-lg">Attributes</h3> <h3 class="font-bold text-xl mt-6">Raw Attributes</h3>
<section> <section>
<strong>Media Profile:</strong> <strong>Media Profile:</strong>
<.subtle_link href={~p"/media_profiles/#{@source.media_profile_id}"}> <.subtle_link href={~p"/media_profiles/#{@source.media_profile_id}"}>
@@ -37,48 +37,18 @@
</div> </div>
</:tab> </:tab>
<:tab title="Pending Media"> <:tab title="Pending Media">
<%= if match?([_|_], @pending_media) do %> <%= live_render(
<h4 class="text-white text-lg mb-6">Shows a maximum of 100 media items</h4> @conn,
<.table rows={@pending_media} table_class="text-black dark:text-white"> Pinchflat.Sources.MediaItemTableLive,
<:col :let={media_item} label="Title"> session: %{"source_id" => @source.id, "media_state" => "pending"}
<.subtle_link href={~p"/sources/#{@source.id}/media/#{media_item.id}"}> ) %>
<%= StringUtils.truncate(media_item.title, 50) %>
</.subtle_link>
</:col>
<:col :let={media_item} label="" class="flex place-content-evenly">
<.icon_link href={~p"/sources/#{@source.id}/media/#{media_item.id}"} icon="hero-eye" class="mx-1" />
<.icon_link
href={~p"/sources/#{@source.id}/media/#{media_item.id}/edit"}
icon="hero-pencil-square"
class="mx-1"
/>
</:col>
</.table>
<% else %>
<p class="text-black dark:text-white">Nothing Here!</p>
<% end %>
</:tab> </:tab>
<:tab title="Downloaded Media"> <:tab title="Downloaded Media">
<%= if match?([_|_], @downloaded_media) do %> <%= live_render(
<h4 class="text-white text-lg mb-6">Shows a maximum of 100 media items (<%= @total_downloaded %> total)</h4> @conn,
<.table rows={@downloaded_media} table_class="text-black dark:text-white"> Pinchflat.Sources.MediaItemTableLive,
<:col :let={media_item} label="Title"> session: %{"source_id" => @source.id, "media_state" => "downloaded"}
<.subtle_link href={~p"/sources/#{@source.id}/media/#{media_item.id}"}> ) %>
<%= StringUtils.truncate(media_item.title, 50) %>
</.subtle_link>
</:col>
<:col :let={media_item} label="" class="flex place-content-evenly">
<.icon_link href={~p"/sources/#{@source.id}/media/#{media_item.id}"} icon="hero-eye" class="mx-1" />
<.icon_link
href={~p"/sources/#{@source.id}/media/#{media_item.id}/edit"}
icon="hero-pencil-square"
class="mx-1"
/>
</:col>
</.table>
<% else %>
<p class="text-black dark:text-white">Nothing Here!</p>
<% end %>
</:tab> </:tab>
<:tab title="Pending Tasks"> <:tab title="Pending Tasks">
<%= if match?([_|_], @pending_tasks) do %> <%= if match?([_|_], @pending_tasks) do %>
@@ -94,7 +64,7 @@
</:col> </:col>
</.table> </.table>
<% else %> <% else %>
<p class="text-black dark:text-white">Nothing Here!</p> <p>Nothing Here!</p>
<% end %> <% end %>
</:tab> </:tab>
</.tabbed_layout> </.tabbed_layout>
@@ -23,7 +23,7 @@
field={f[:custom_name]} field={f[:custom_name]}
type="text" type="text"
label="Custom Name" label="Custom Name"
help="Something descriptive. Does not impact indexing or downloading" help="Does not impact indexing or downloading. Will be inferred from the source if left blank"
/> />
<.input field={f[:original_url]} type="text" label="Source URL" help="URL of a channel or playlist (required)" /> <.input field={f[:original_url]} type="text" label="Source URL" help="URL of a channel or playlist (required)" />
@@ -111,7 +111,8 @@
type="text" type="text"
label="Title Filter Regex" label="Title Filter Regex"
placeholder="(?i)^How to Bike$" placeholder="(?i)^How to Bike$"
help="A PCRE-compatible regex. Only media with titles that match this regex will be downloaded. Look up 'SQLean Regex docs' for more" help={title_filter_regex_help()}
html_help={true}
/> />
<section <section
+2
View File
@@ -35,6 +35,7 @@ defmodule PinchflatWeb.Router do
resources "/sources", Sources.SourceController do resources "/sources", Sources.SourceController do
post "/force_download", Sources.SourceController, :force_download post "/force_download", Sources.SourceController, :force_download
post "/force_index", Sources.SourceController, :force_index post "/force_index", Sources.SourceController, :force_index
post "/force_metadata_refresh", Sources.SourceController, :force_metadata_refresh
resources "/media", MediaItems.MediaItemController, only: [:show, :edit, :update, :delete] do resources "/media", MediaItems.MediaItemController, only: [:show, :edit, :update, :delete] do
post "/force_download", MediaItems.MediaItemController, :force_download post "/force_download", MediaItems.MediaItemController, :force_download
@@ -49,6 +50,7 @@ defmodule PinchflatWeb.Router do
get "/sources/:uuid/feed", Podcasts.PodcastController, :rss_feed get "/sources/:uuid/feed", Podcasts.PodcastController, :rss_feed
get "/sources/:uuid/feed_image", Podcasts.PodcastController, :feed_image get "/sources/:uuid/feed_image", Podcasts.PodcastController, :feed_image
get "/media/:uuid/episode_image", Podcasts.PodcastController, :episode_image
get "/media/:uuid/stream", MediaItems.MediaItemController, :stream get "/media/:uuid/stream", MediaItems.MediaItemController, :stream
end end
+1 -1
View File
@@ -4,7 +4,7 @@ defmodule Pinchflat.MixProject do
def project do def project do
[ [
app: :pinchflat, app: :pinchflat,
version: "0.1.14", version: "0.1.17",
elixir: "~> 1.16", elixir: "~> 1.16",
elixirc_paths: elixirc_paths(Mix.env()), elixirc_paths: elixirc_paths(Mix.env()),
start_permanent: Mix.env() == :prod, start_permanent: Mix.env() == :prod,
@@ -0,0 +1,11 @@
defmodule Pinchflat.Repo.Migrations.AddUploadDateIndexToMediaItems do
use Ecto.Migration
def change do
alter table(:media_items) do
add :upload_date_index, :integer, null: false, default: 0
end
create index("media_items", [:upload_date])
end
end
+1 -1
View File
@@ -79,7 +79,7 @@ ARG PORT=8945
RUN apt-get update -y RUN apt-get update -y
RUN apt-get install -y libstdc++6 openssl libncurses5 locales ca-certificates \ RUN apt-get install -y libstdc++6 openssl libncurses5 locales ca-certificates \
ffmpeg curl git openssh-client nano python3 python3-pip ffmpeg curl git openssh-client nano python3 python3-pip jq procps
RUN apt-get clean && rm -f /var/lib/apt/lists/*_* RUN apt-get clean && rm -f /var/lib/apt/lists/*_*
# Download and update YT-DLP # Download and update YT-DLP
@@ -53,6 +53,30 @@ defmodule Pinchflat.Boot.PreJobStartupTasksTest do
end end
end end
describe "create_blank_user_script_file" do
test "creates a blank script file" do
base_dir = Application.get_env(:pinchflat, :extras_directory)
filepath = Path.join([base_dir, "user-scripts", "lifecycle"])
File.rm(filepath)
refute File.exists?(filepath)
PreJobStartupTasks.init(%{})
assert File.exists?(filepath)
end
test "gives it 755 permissions" do
base_dir = Application.get_env(:pinchflat, :extras_directory)
filepath = Path.join([base_dir, "user-scripts", "lifecycle"])
File.rm(filepath)
PreJobStartupTasks.init(%{})
assert File.stat!(filepath).mode == 0o100755
end
end
describe "apply_default_settings" do describe "apply_default_settings" do
test "sets yt_dlp version" do test "sets yt_dlp version" do
Settings.set(yt_dlp_version: nil) Settings.set(yt_dlp_version: nil)
@@ -33,6 +33,15 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilderTest do
assert {:output, "/tmp/test/media/#{media_item.source.custom_name}.%(ext)s"} in res assert {:output, "/tmp/test/media/#{media_item.source.custom_name}.%(ext)s"} in res
end end
test "respects custom media_item-related output path options", %{media_item: media_item} do
media_item =
update_media_profile_attribute(media_item, %{output_path_template: "{{ media_upload_date_index }}.%(ext)s"})
assert {:ok, res} = DownloadOptionBuilder.build(media_item)
assert {:output, "/tmp/test/media/99.%(ext)s"} in res
end
test "uses source's output override if present", %{media_item: media_item} do test "uses source's output override if present", %{media_item: media_item} do
source = media_item.source source = media_item.source
{:ok, _} = Sources.update_source(source, %{output_path_template_override: "override.%(ext)s"}) {:ok, _} = Sources.update_source(source, %{output_path_template_override: "override.%(ext)s"})
@@ -53,7 +62,6 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilderTest do
assert {:ok, res} = DownloadOptionBuilder.build(media_item) assert {:ok, res} = DownloadOptionBuilder.build(media_item)
assert :no_progress in res assert :no_progress in res
assert :windows_filenames in res
assert :force_overwrites in res assert :force_overwrites in res
assert {:parse_metadata, "%(upload_date>%Y-%m-%d)s:(?P<meta_date>.+)"} in res assert {:parse_metadata, "%(upload_date>%Y-%m-%d)s:(?P<meta_date>.+)"} in res
end end
@@ -251,7 +259,7 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilderTest do
assert {:ok, res} = DownloadOptionBuilder.build(media_item) assert {:ok, res} = DownloadOptionBuilder.build(media_item)
assert :extract_audio in res assert :extract_audio in res
assert {:format, "bestaudio[ext=m4a]"} in res assert {:format, "bestaudio[ext=m4a]/bestaudio[ext=mp3]/bestaudio/best[ext=m4a]/best[ext=mp3]/best"} in res
refute {:remux_video, "mp4"} in res refute {:remux_video, "mp4"} in res
end end
@@ -387,6 +395,12 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilderTest do
end end
describe "build_output_path_for/1" do describe "build_output_path_for/1" do
test "builds an output path for a media item", %{media_item: media_item} do
path = DownloadOptionBuilder.build_output_path_for(media_item)
assert path == "/tmp/test/media/%(title)S.%(ext)s"
end
test "builds an output path for a source", %{media_item: media_item} do test "builds an output path for a source", %{media_item: media_item} do
path = DownloadOptionBuilder.build_output_path_for(media_item.source) path = DownloadOptionBuilder.build_output_path_for(media_item.source)
@@ -12,9 +12,8 @@ defmodule Pinchflat.Downloading.MediaDownloadWorkerTest do
setup :verify_on_exit! setup :verify_on_exit!
setup do setup do
stub(HTTPClientMock, :get, fn _url, _headers, _opts -> stub(UserScriptRunnerMock, :run, fn _event_type, _data -> :ok end)
{:ok, ""} stub(HTTPClientMock, :get, fn _url, _headers, _opts -> {:ok, ""} end)
end)
media_item = media_item =
%{media_filepath: nil} %{media_filepath: nil}
@@ -106,6 +105,18 @@ defmodule Pinchflat.Downloading.MediaDownloadWorkerTest do
end) end)
end end
test "does not set the job to retryable if retrying wouldn't fix the issue", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl ->
{:error, "Something something Video unavailable something something", 1}
end)
Oban.Testing.with_testing_mode(:inline, fn ->
{:ok, job} = Oban.insert(MediaDownloadWorker.new(%{id: media_item.id, redownload?: true}))
assert job.state == "completed"
end)
end
test "it ensures error are returned in a 2-item tuple", %{media_item: media_item} do test "it ensures error are returned in a 2-item tuple", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl -> {:error, "error", 1} end) expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl -> {:error, "error", 1} end)
@@ -173,6 +184,20 @@ defmodule Pinchflat.Downloading.MediaDownloadWorkerTest do
assert media_item.media_redownloaded_at == nil assert media_item.media_redownloaded_at == nil
end end
test "calls the user script runner", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl ->
{:ok, render_metadata(:media_metadata)}
end)
expect(UserScriptRunnerMock, :run, fn :media_downloaded, data ->
assert data.id == media_item.id
:ok
end)
perform_job(MediaDownloadWorker, %{id: media_item.id})
end
test "does not blow up if the record doesn't exist" do test "does not blow up if the record doesn't exist" do
assert :ok = perform_job(MediaDownloadWorker, %{id: 0}) assert :ok = perform_job(MediaDownloadWorker, %{id: 0})
end end
@@ -1,13 +1,22 @@
defmodule Pinchflat.Downloading.MediaRetentionWorkerTest do defmodule Pinchflat.Downloading.MediaRetentionWorkerTest do
use Pinchflat.DataCase use Pinchflat.DataCase
import Mox
import Pinchflat.MediaFixtures import Pinchflat.MediaFixtures
import Pinchflat.SourcesFixtures import Pinchflat.SourcesFixtures
alias Pinchflat.Media alias Pinchflat.Media
alias Pinchflat.Downloading.MediaRetentionWorker alias Pinchflat.Downloading.MediaRetentionWorker
setup :verify_on_exit!
describe "perform/1" do describe "perform/1" do
setup do
stub(UserScriptRunnerMock, :run, fn _event_type, _data -> :ok end)
:ok
end
test "deletes media files that are past their retention date" do test "deletes media files that are past their retention date" do
{_source, old_media_item, new_media_item} = prepare_records() {_source, old_media_item, new_media_item} = prepare_records()
@@ -1,5 +1,5 @@
defmodule Pinchflat.Downloading.OutputPath.ParserTest do defmodule Pinchflat.Downloading.OutputPath.ParserTest do
use ExUnit.Case, async: true use ExUnit.Case, async: false
alias Pinchflat.Downloading.OutputPath.Parser alias Pinchflat.Downloading.OutputPath.Parser
@@ -27,5 +27,15 @@ defmodule Pinchflat.Downloading.OutputPathBuilderTest do
assert res == "/videos/%(title)s.%(ext)s" assert res == "/videos/%(title)s.%(ext)s"
end end
test "recursively expands variables" do
additional_options = %{
"media_upload_date_index" => "99"
}
assert {:ok, res} = OutputPathBuilder.build("{{ season_episode_index_from_date }}.{{ ext }}", additional_options)
assert res == "s%(upload_date>%Y)Se%(upload_date>%m%d)S99.%(ext)S"
end
end end
end end
@@ -89,5 +89,25 @@ defmodule Pinchflat.FastIndexing.FastIndexingHelpersTest do
refute_enqueued(worker: MediaDownloadWorker) refute_enqueued(worker: MediaDownloadWorker)
end end
test "does not blow up if a media item cannot be created", %{source: source} do
expect(HTTPClientMock, :get, fn _url -> {:ok, "<yt:videoId>test_1</yt:videoId>"} end)
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot ->
{:ok, "{}"}
end)
assert [] = FastIndexingHelpers.kickoff_download_tasks_from_youtube_rss_feed(source)
end
test "does not blow up if a media item causes a yt-dlp error", %{source: source} do
expect(HTTPClientMock, :get, fn _url -> {:ok, "<yt:videoId>test_1</yt:videoId>"} end)
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot ->
{:error, "message", 1}
end)
assert [] = FastIndexingHelpers.kickoff_download_tasks_from_youtube_rss_feed(source)
end
end end
end end
@@ -1,7 +1,7 @@
defmodule Pinchflat.Notifications.CommandRunnerTest do defmodule Pinchflat.Lifecycle.Notifications.CommandRunnerTest do
use ExUnit.Case, async: true use ExUnit.Case, async: false
alias Pinchflat.Notifications.CommandRunner, as: Runner alias Pinchflat.Lifecycle.Notifications.CommandRunner, as: Runner
@original_executable Application.compile_env(:pinchflat, :apprise_executable) @original_executable Application.compile_env(:pinchflat, :apprise_executable)
@@ -1,11 +1,11 @@
defmodule Pinchflat.Notifications.SourceNotificationsTest do defmodule Pinchflat.Lifecycle.Notifications.SourceNotificationsTest do
use Pinchflat.DataCase use Pinchflat.DataCase
import Mox import Mox
import Pinchflat.MediaFixtures import Pinchflat.MediaFixtures
import Pinchflat.SourcesFixtures import Pinchflat.SourcesFixtures
alias Pinchflat.Notifications.SourceNotifications alias Pinchflat.Lifecycle.Notifications.SourceNotifications
@apprise_servers ["server_1", "server_2"] @apprise_servers ["server_1", "server_2"]
@@ -60,6 +60,17 @@ defmodule Pinchflat.Notifications.SourceNotificationsTest do
end) end)
end end
test "does not send a notification if the source is set to not download media" do
source = source_fixture(%{download_media: false})
expect(AppriseRunnerMock, :run, 0, fn _, _ -> {:ok, ""} end)
SourceNotifications.wrap_new_media_notification(@apprise_servers, source, fn ->
media_item_fixture(%{source_id: source.id, media_filepath: nil})
media_item_fixture(%{source_id: source.id, media_filepath: "file.mp4"})
end)
end
test "returns the value of the function" do test "returns the value of the function" do
source = source_fixture() source = source_fixture()
expect(AppriseRunnerMock, :run, 0, fn _, _ -> {:ok, ""} end) expect(AppriseRunnerMock, :run, 0, fn _, _ -> {:ok, ""} end)
@@ -0,0 +1,79 @@
defmodule Pinchflat.Lifecycle.UserScripts.CommandRunnerTest do
use ExUnit.Case, async: false
alias Pinchflat.Utils.FilesystemUtils
alias Pinchflat.Lifecycle.UserScripts.CommandRunner, as: Runner
setup do
FilesystemUtils.write_p!(filepath(), "")
File.chmod(filepath(), 0o755)
:ok
end
describe "run/2" do
test "runs the provided lifecycle file if present" do
# We *love* indirectly testing side effects
tmp_dir = Application.get_env(:pinchflat, :tmpfile_directory)
File.write(filepath(), "#!/bin/bash\ntouch #{tmp_dir}/test_file\n")
refute File.exists?("#{tmp_dir}/test_file")
assert :ok = Runner.run(:media_downloaded, %{})
assert File.exists?("#{tmp_dir}/test_file")
end
test "passes the event name to the script" do
tmp_dir = Application.get_env(:pinchflat, :tmpfile_directory)
File.write(filepath(), "#!/bin/bash\necho $1 > #{tmp_dir}/event_name\n")
assert :ok = Runner.run(:media_downloaded, %{})
assert File.read!("#{tmp_dir}/event_name") == "media_downloaded\n"
end
test "passes the encoded data to the script" do
tmp_dir = Application.get_env(:pinchflat, :tmpfile_directory)
File.write(filepath(), "#!/bin/bash\necho $2 > #{tmp_dir}/encoded_data\n")
assert :ok = Runner.run(:media_downloaded, %{foo: "bar"})
assert File.read!("#{tmp_dir}/encoded_data") == "{\"foo\":\"bar\"}\n"
end
test "does nothing if the lifecycle file is not present" do
:ok = File.rm(filepath())
assert :ok = Runner.run(:media_downloaded, %{})
end
test "does nothing if the lifecycle file is empty" do
File.write(filepath(), "")
assert :ok = Runner.run(:media_downloaded, %{})
end
test "returns :ok if the command exits with a non-zero status" do
File.write(filepath(), "#!/bin/bash\nexit 1\n")
assert :ok = Runner.run(:media_downloaded, %{})
end
test "gets upset if you pass an invalid event type" do
assert_raise ArgumentError, "Invalid event type: :invalid_event", fn ->
Runner.run(:invalid_event, %{})
end
end
test "gets upset if the record cannot be decoded" do
File.write(filepath(), "#!/bin/bash")
assert_raise MatchError, fn ->
Runner.run(:media_downloaded, %Ecto.Changeset{})
end
end
end
defp filepath do
base_dir = Application.get_env(:pinchflat, :extras_directory)
Path.join([base_dir, "user-scripts", "lifecycle"])
end
end
+186
View File
@@ -29,6 +29,156 @@ defmodule Pinchflat.MediaTest do
Repo.reload!(metadata) Repo.reload!(metadata)
end end
end end
test "can be JSON encoded without error" do
media_item = media_item_fixture()
assert {:ok, _} = Phoenix.json_library().encode(media_item)
end
end
describe "schema when testing upload_date_index and source is a channel" do
test "upload_date_index is set to 99 if it's the only video uploaded that day" do
upload_date = Date.utc_today()
source = source_fixture(%{collection_type: :channel})
media_item = media_item_fixture(%{source_id: source.id, upload_date: upload_date})
assert media_item.upload_date_index == 99
end
test "upload_date_index is set to 98 if it's the second video uploaded that day" do
upload_date = Date.utc_today()
source = source_fixture(%{collection_type: :channel})
media_item_one = media_item_fixture(%{source_id: source.id, upload_date: upload_date})
media_item_two = media_item_fixture(%{source_id: source.id, upload_date: upload_date})
assert media_item_one.upload_date_index == 99
assert media_item_two.upload_date_index == 98
end
test "upload_date_index doesn't decrement if the video is uploaded on a different day" do
today = Date.utc_today()
one_day_ago = Date.add(today, -1)
source = source_fixture(%{collection_type: :channel})
media_item_new = media_item_fixture(%{source_id: source.id, upload_date: today})
media_item_old = media_item_fixture(%{source_id: source.id, upload_date: one_day_ago})
assert media_item_new.upload_date_index == 99
assert media_item_old.upload_date_index == 99
end
test "recomputes upload_date_index if an upload_date is changed...somehow" do
today = Date.utc_today()
one_day_ago = Date.add(today, -1)
source = source_fixture(%{collection_type: :channel})
media_item_new = media_item_fixture(%{source_id: source.id, upload_date: today})
media_item_old = media_item_fixture(%{source_id: source.id, upload_date: one_day_ago})
{:ok, updated_media_item} = Media.update_media_item(media_item_old, %{upload_date: today})
assert media_item_new.upload_date_index == 99
assert updated_media_item.upload_date_index == 98
end
test "upload_date_index doesn't decrement if the video is for a different source" do
today = Date.utc_today()
source_one = source_fixture(%{collection_type: :channel})
source_two = source_fixture(%{collection_type: :channel})
media_item_one = media_item_fixture(%{source_id: source_one.id, upload_date: today})
media_item_two = media_item_fixture(%{source_id: source_two.id, upload_date: today})
assert media_item_one.upload_date_index == 99
assert media_item_two.upload_date_index == 99
end
test "upload_date_index doesn't decrement if the a video's upload_date is updated but doesn't change" do
today = Date.utc_today()
source = source_fixture(%{collection_type: :channel})
media_item_one = media_item_fixture(%{source_id: source.id, upload_date: today})
_media_item_two = media_item_fixture(%{source_id: source.id, upload_date: today})
{:ok, updated_media_item} = Media.update_media_item(media_item_one, %{upload_date: today, title: "New title"})
assert updated_media_item.upload_date_index == 99
end
end
describe "schema when testing upload_date_index and source is a playlist" do
test "upload_date_index is set to 0 if it's the only video uploaded that day" do
upload_date = Date.utc_today()
source = source_fixture(%{collection_type: :playlist})
media_item = media_item_fixture(%{source_id: source.id, upload_date: upload_date})
assert media_item.upload_date_index == 0
end
test "upload_date_index is set to 1 if it's the second video uploaded that day" do
upload_date = Date.utc_today()
source = source_fixture(%{collection_type: :playlist})
media_item_one = media_item_fixture(%{source_id: source.id, upload_date: upload_date})
media_item_two = media_item_fixture(%{source_id: source.id, upload_date: upload_date})
assert media_item_one.upload_date_index == 0
assert media_item_two.upload_date_index == 1
end
test "upload_date_index doesn't increment if the video is uploaded on a different day" do
today = Date.utc_today()
one_day_ago = Date.add(today, -1)
source = source_fixture(%{collection_type: :playlist})
media_item_new = media_item_fixture(%{source_id: source.id, upload_date: today})
media_item_old = media_item_fixture(%{source_id: source.id, upload_date: one_day_ago})
assert media_item_new.upload_date_index == 0
assert media_item_old.upload_date_index == 0
end
test "recomputes upload_date_index if an upload_date is changed...somehow" do
today = Date.utc_today()
one_day_ago = Date.add(today, -1)
source = source_fixture(%{collection_type: :playlist})
media_item_new = media_item_fixture(%{source_id: source.id, upload_date: today})
media_item_old = media_item_fixture(%{source_id: source.id, upload_date: one_day_ago})
{:ok, updated_media_item} = Media.update_media_item(media_item_old, %{upload_date: today})
assert media_item_new.upload_date_index == 0
assert updated_media_item.upload_date_index == 1
end
test "upload_date_index doesn't increment if the video is for a different source" do
today = Date.utc_today()
source_one = source_fixture(%{collection_type: :playlist})
source_two = source_fixture(%{collection_type: :playlist})
media_item_one = media_item_fixture(%{source_id: source_one.id, upload_date: today})
media_item_two = media_item_fixture(%{source_id: source_two.id, upload_date: today})
assert media_item_one.upload_date_index == 0
assert media_item_two.upload_date_index == 0
end
test "upload_date_index doesn't increment if the a video's upload_date is updated but doesn't change" do
today = Date.utc_today()
source = source_fixture(%{collection_type: :playlist})
media_item_one = media_item_fixture(%{source_id: source.id, upload_date: today})
_media_item_two = media_item_fixture(%{source_id: source.id, upload_date: today})
{:ok, updated_media_item} = Media.update_media_item(media_item_one, %{upload_date: today, title: "New title"})
assert updated_media_item.upload_date_index == 0
end
end end
describe "list_media_items/0" do describe "list_media_items/0" do
@@ -743,6 +893,12 @@ defmodule Pinchflat.MediaTest do
end end
describe "delete_media_item/2 when testing file deletion" do describe "delete_media_item/2 when testing file deletion" do
setup do
stub(UserScriptRunnerMock, :run, fn _event_type, _data -> :ok end)
:ok
end
test "deletes the media item's files" do test "deletes the media item's files" do
media_item = media_item_with_attachments() media_item = media_item_with_attachments()
@@ -793,9 +949,27 @@ defmodule Pinchflat.MediaTest do
:ok = File.rm(Path.join([root_directory, "test.txt"])) :ok = File.rm(Path.join([root_directory, "test.txt"]))
:ok = File.rmdir(root_directory) :ok = File.rmdir(root_directory)
end end
test "calls the user script runner" do
media_item = media_item_with_attachments()
expect(UserScriptRunnerMock, :run, fn :media_deleted, data ->
assert data.id == media_item.id
:ok
end)
assert {:ok, _} = Media.delete_media_item(media_item, delete_files: true)
end
end end
describe "delete_media_files/2" do describe "delete_media_files/2" do
setup do
stub(UserScriptRunnerMock, :run, fn _event_type, _data -> :ok end)
:ok
end
test "does not delete the media_item" do test "does not delete the media_item" do
media_item = media_item_fixture() media_item = media_item_fixture()
@@ -848,6 +1022,18 @@ defmodule Pinchflat.MediaTest do
assert {:ok, updated_media_item} = Media.delete_media_files(media_item, %{prevent_download: true}) assert {:ok, updated_media_item} = Media.delete_media_files(media_item, %{prevent_download: true})
assert updated_media_item.prevent_download assert updated_media_item.prevent_download
end end
test "calls the user script runner" do
media_item = media_item_with_attachments()
expect(UserScriptRunnerMock, :run, fn :media_deleted, data ->
assert data.id == media_item.id
:ok
end)
assert {:ok, _} = Media.delete_media_files(media_item)
end
end end
describe "change_media_item/1" do describe "change_media_item/1" do
@@ -89,6 +89,48 @@ defmodule Pinchflat.Metadata.MetadataFileHelpersTest do
assert filepath =~ ~r{/media_items/#{media_item.id}/img_2.jpg} assert filepath =~ ~r{/media_items/#{media_item.id}/img_2.jpg}
end end
test "will fall back to a non-jpg if it has to", %{media_item: media_item} do
metadata = %{
"thumbnails" => [
%{"url" => "https://i.ytimg.com/vi/ABC123/img_1.webp", "preference" => -1}
]
}
filepath = Helpers.download_and_store_thumbnail_for(media_item, metadata)
assert filepath =~ ~r{/media_items/#{media_item.id}/img_1.webp}
end
test "does not require a preference field", %{media_item: media_item} do
metadata = %{
"thumbnails" => [
%{"url" => "https://i.ytimg.com/vi/ABC123/img_1.webp"}
]
}
filepath = Helpers.download_and_store_thumbnail_for(media_item, metadata)
assert filepath =~ ~r{/media_items/#{media_item.id}/img_1.webp}
end
end
describe "download_and_store_thumbnail_for/2 when not downloading thumbnails" do
test "returns nil if there are no thumbnails", %{media_item: media_item} do
metadata = %{"thumbnails" => []}
filepath = Helpers.download_and_store_thumbnail_for(media_item, metadata)
assert filepath == nil
end
test "returns nil if there is no thumbnail field", %{media_item: media_item} do
metadata = %{}
filepath = Helpers.download_and_store_thumbnail_for(media_item, metadata)
assert filepath == nil
end
end end
describe "parse_upload_date/1" do describe "parse_upload_date/1" do
@@ -46,6 +46,14 @@ defmodule Pinchflat.Metadata.MetadataParserTest do
assert result.livestream == metadata["was_live"] assert result.livestream == metadata["was_live"]
end end
test "the livestream flag defaults to false", %{metadata: metadata} do
metadata = Map.put(metadata, "was_live", nil)
result = Parser.parse_for_media_item(metadata)
assert result.livestream == false
end
test "it extracts the duration in seconds", %{metadata: metadata} do test "it extracts the duration in seconds", %{metadata: metadata} do
result = Parser.parse_for_media_item(metadata) result = Parser.parse_for_media_item(metadata)
@@ -106,7 +114,7 @@ defmodule Pinchflat.Metadata.MetadataParserTest do
:ok = File.cp(thumbnail_filepath_fixture(), thumbnail_filepath) :ok = File.cp(thumbnail_filepath_fixture(), thumbnail_filepath)
on_exit(fn -> File.rm(thumbnail_filepath) end) on_exit(fn -> File.rm_rf(thumbnail_filepath) end)
{:ok, filepath: thumbnail_filepath} {:ok, filepath: thumbnail_filepath}
end end
@@ -127,7 +135,7 @@ defmodule Pinchflat.Metadata.MetadataParserTest do
end end
test "doesn't include thumbnail if the file doesn't exist on-disk", %{metadata: metadata, filepath: filepath} do test "doesn't include thumbnail if the file doesn't exist on-disk", %{metadata: metadata, filepath: filepath} do
File.rm(filepath) File.rm_rf(filepath)
result = Parser.parse_for_media_item(metadata) result = Parser.parse_for_media_item(metadata)
@@ -156,7 +164,7 @@ defmodule Pinchflat.Metadata.MetadataParserTest do
infojson_filepath = metadata["infojson_filename"] infojson_filepath = metadata["infojson_filename"]
:ok = File.cp(infojson_filepath_fixture(), infojson_filepath) :ok = File.cp(infojson_filepath_fixture(), infojson_filepath)
on_exit(fn -> File.rm(infojson_filepath) end) on_exit(fn -> File.rm_rf(infojson_filepath) end)
{:ok, filepath: infojson_filepath} {:ok, filepath: infojson_filepath}
end end
@@ -168,7 +176,7 @@ defmodule Pinchflat.Metadata.MetadataParserTest do
end end
test "doesn't include metadata if the file doesn't exist on-disk", %{metadata: metadata, filepath: filepath} do test "doesn't include metadata if the file doesn't exist on-disk", %{metadata: metadata, filepath: filepath} do
File.rm(filepath) File.rm_rf(filepath)
result = Parser.parse_for_media_item(metadata) result = Parser.parse_for_media_item(metadata)
@@ -137,6 +137,34 @@ defmodule Pinchflat.Podcasts.RssFeedBuilderTest do
assert String.contains?(item_xml, ~s(length="1234")) assert String.contains?(item_xml, ~s(length="1234"))
assert String.contains?(item_xml, ~s(type="video/mp4")) assert String.contains?(item_xml, ~s(type="video/mp4"))
end end
test "returns image tags if the media has a thumbnail", %{source: source} do
media_item = media_item_with_attachments(%{source_id: source.id, media_size_bytes: 1234})
res = RssFeedBuilder.build(source)
[_before, item_xml, _after] = String.split(res, ~r(</?item>))
assert String.contains?(
item_xml,
~s(<itunes:image href="http://localhost:8945/media/#{media_item.uuid}/episode_image.jpg"></itunes:image>)
)
assert String.contains?(
item_xml,
~s(<podcast:images srcset="http://localhost:8945/media/#{media_item.uuid}/episode_image.jpg" />)
)
end
test "does not return image tags if the media does not have a thumbnail", %{source: source} do
media_item = media_item_with_attachments(%{source_id: source.id})
File.rm!(media_item.thumbnail_filepath)
res = RssFeedBuilder.build(source)
[_before, item_xml, _after] = String.split(res, ~r(</?item>))
refute String.contains?(item_xml, ~s(itunes:image))
refute String.contains?(item_xml, ~s(podcast:images))
end
end end
defp format_date(date) do defp format_date(date) do
+17
View File
@@ -1,6 +1,7 @@
defmodule Pinchflat.ProfilesTest do defmodule Pinchflat.ProfilesTest do
use Pinchflat.DataCase use Pinchflat.DataCase
import Mox
import Pinchflat.MediaFixtures import Pinchflat.MediaFixtures
import Pinchflat.SourcesFixtures import Pinchflat.SourcesFixtures
import Pinchflat.ProfilesFixtures import Pinchflat.ProfilesFixtures
@@ -10,6 +11,16 @@ defmodule Pinchflat.ProfilesTest do
@invalid_attrs %{name: nil, output_path_template: nil} @invalid_attrs %{name: nil, output_path_template: nil}
setup :verify_on_exit!
describe "schema" do
test "can be JSON encoded without error" do
profile = media_profile_fixture()
assert {:ok, _} = Phoenix.json_library().encode(profile)
end
end
describe "list_media_profiles/0" do describe "list_media_profiles/0" do
test "it returns all media_profiles" do test "it returns all media_profiles" do
media_profile = media_profile_fixture() media_profile = media_profile_fixture()
@@ -104,6 +115,12 @@ defmodule Pinchflat.ProfilesTest do
end end
describe "delete_media_profile/2 when deleting files" do describe "delete_media_profile/2 when deleting files" do
setup do
stub(UserScriptRunnerMock, :run, fn _event_type, _data -> :ok end)
:ok
end
test "still deletes all the needful records" do test "still deletes all the needful records" do
media_profile = media_profile_fixture() media_profile = media_profile_fixture()
source = source_fixture(media_profile_id: media_profile.id) source = source_fixture(media_profile_id: media_profile.id)
@@ -1,5 +1,5 @@
defmodule Pinchflat.SlowIndexing.FileFollowerServerTest do defmodule Pinchflat.SlowIndexing.FileFollowerServerTest do
use ExUnit.Case, async: true use ExUnit.Case, async: false
alias alias Pinchflat.Utils.FilesystemUtils alias alias Pinchflat.Utils.FilesystemUtils
alias Pinchflat.SlowIndexing.FileFollowerServer alias Pinchflat.SlowIndexing.FileFollowerServer
+36 -2
View File
@@ -32,6 +32,12 @@ defmodule Pinchflat.SourcesTest do
Repo.reload!(metadata) Repo.reload!(metadata)
end end
end end
test "can be JSON encoded without error" do
source = source_fixture()
assert {:ok, _} = Phoenix.json_library().encode(source)
end
end end
describe "output_path_template/1" do describe "output_path_template/1" do
@@ -131,6 +137,18 @@ defmodule Pinchflat.SourcesTest do
assert String.starts_with?(source.collection_id, "some_playlist_id_") assert String.starts_with?(source.collection_id, "some_playlist_id_")
end end
test "adds an error if the runner fails" do
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot -> {:error, "some error", 1} end)
valid_attrs = %{
media_profile_id: media_profile_fixture().id,
original_url: "https://www.youtube.com/channel/abc123"
}
assert {:error, %Ecto.Changeset{} = changeset} = Sources.create_source(valid_attrs)
assert "could not fetch source details from URL" in errors_on(changeset).original_url
end
test "you can specify a custom custom_name" do test "you can specify a custom custom_name" do
expect(YtDlpRunnerMock, :run, &channel_mock/3) expect(YtDlpRunnerMock, :run, &channel_mock/3)
@@ -502,14 +520,24 @@ defmodule Pinchflat.SourcesTest do
assert source.index_frequency_minutes == 0 assert source.index_frequency_minutes == 0
end end
test "updating will kickoff a metadata storage worker" do test "updating will kickoff a metadata storage worker if the original_url changes" do
expect(YtDlpRunnerMock, :run, &playlist_mock/3)
source = source_fixture() source = source_fixture()
update_attrs = %{name: "some updated name"} update_attrs = %{original_url: "https://www.youtube.com/channel/cba321"}
assert {:ok, %Source{} = source} = Sources.update_source(source, update_attrs) assert {:ok, %Source{} = source} = Sources.update_source(source, update_attrs)
assert_enqueued(worker: SourceMetadataStorageWorker, args: %{"id" => source.id}) assert_enqueued(worker: SourceMetadataStorageWorker, args: %{"id" => source.id})
end end
test "updating will not kickoff a metadata storage worker other attrs change" do
source = source_fixture()
update_attrs = %{name: "some new name"}
assert {:ok, %Source{}} = Sources.update_source(source, update_attrs)
refute_enqueued(worker: SourceMetadataStorageWorker)
end
end end
describe "update_source/3 when testing options" do describe "update_source/3 when testing options" do
@@ -590,6 +618,12 @@ defmodule Pinchflat.SourcesTest do
end end
describe "delete_source/2 when deleting files" do describe "delete_source/2 when deleting files" do
setup do
stub(UserScriptRunnerMock, :run, fn _event_type, _data -> :ok end)
:ok
end
test "deletes source and media_items" do test "deletes source and media_items" do
source = source_fixture() source = source_fixture()
media_item = media_item_with_attachments(%{source_id: source.id}) media_item = media_item_with_attachments(%{source_id: source.id})
@@ -1,5 +1,5 @@
defmodule Pinchflat.Utils.ChangesetUtilsTest do defmodule Pinchflat.Utils.ChangesetUtilsTest do
use ExUnit.Case, async: true use ExUnit.Case, async: false
defmodule MockSchema do defmodule MockSchema do
use Ecto.Schema use Ecto.Schema
+1 -1
View File
@@ -1,5 +1,5 @@
defmodule Pinchflat.Utils.CliUtilsTest do defmodule Pinchflat.Utils.CliUtilsTest do
use ExUnit.Case, async: true use ExUnit.Case, async: false
alias Pinchflat.Utils.CliUtils alias Pinchflat.Utils.CliUtils
+1 -1
View File
@@ -1,5 +1,5 @@
defmodule Pinchflat.Utils.DatetimeUtilsTest do defmodule Pinchflat.Utils.DatetimeUtilsTest do
use ExUnit.Case, async: true use ExUnit.Case, async: false
alias Pinchflat.Utils.DatetimeUtils alias Pinchflat.Utils.DatetimeUtils
@@ -5,6 +5,38 @@ defmodule Pinchflat.Utils.FilesystemUtilsTest do
alias Pinchflat.Utils.FilesystemUtils alias Pinchflat.Utils.FilesystemUtils
describe "exists_and_nonempty?" do
test "returns true if a file exists and has contents" do
filepath = FilesystemUtils.generate_metadata_tmpfile(:json)
File.write(filepath, "{}")
assert FilesystemUtils.exists_and_nonempty?(filepath)
File.rm!(filepath)
end
test "returns false if a file doesn't exist" do
refute FilesystemUtils.exists_and_nonempty?("/nonexistent/file.json")
end
test "returns false if a file exists but is empty" do
filepath = FilesystemUtils.generate_metadata_tmpfile(:json)
refute FilesystemUtils.exists_and_nonempty?(filepath)
File.rm!(filepath)
end
test "trims the contents before checking" do
filepath = FilesystemUtils.generate_metadata_tmpfile(:json)
File.write(filepath, " \n\n \r\n ")
refute FilesystemUtils.exists_and_nonempty?(filepath)
File.rm!(filepath)
end
end
describe "generate_metadata_tmpfile/1" do describe "generate_metadata_tmpfile/1" do
test "creates a tmpfile and returns its path" do test "creates a tmpfile and returns its path" do
res = FilesystemUtils.generate_metadata_tmpfile(:json) res = FilesystemUtils.generate_metadata_tmpfile(:json)
+1 -1
View File
@@ -1,5 +1,5 @@
defmodule Pinchflat.Utils.FunctionUtilsTest do defmodule Pinchflat.Utils.FunctionUtilsTest do
use ExUnit.Case, async: true use ExUnit.Case, async: false
alias Pinchflat.Utils.FunctionUtils alias Pinchflat.Utils.FunctionUtils
@@ -0,0 +1,19 @@
defmodule Pinchflat.Utils.NumberUtilsTest do
use ExUnit.Case, async: false
alias Pinchflat.Utils.NumberUtils
describe "clamp/3" do
test "returns the minimum when the number is less than the minimum" do
assert NumberUtils.clamp(1, 2, 3) == 2
end
test "returns the maximum when the number is greater than the maximum" do
assert NumberUtils.clamp(4, 2, 3) == 3
end
test "returns the number when it is between the minimum and maximum" do
assert NumberUtils.clamp(2, 1, 3) == 2
end
end
end
+1 -1
View File
@@ -1,5 +1,5 @@
defmodule Pinchflat.Utils.StringUtilsTest do defmodule Pinchflat.Utils.StringUtilsTest do
use ExUnit.Case, async: true use ExUnit.Case, async: false
alias Pinchflat.Utils.StringUtils alias Pinchflat.Utils.StringUtils
+1 -1
View File
@@ -1,5 +1,5 @@
defmodule Pinchflat.Utils.XmlUtilsTest do defmodule Pinchflat.Utils.XmlUtilsTest do
use ExUnit.Case, async: true use ExUnit.Case, async: false
alias Pinchflat.Utils.XmlUtils alias Pinchflat.Utils.XmlUtils
+12 -1
View File
@@ -1,5 +1,5 @@
defmodule Pinchflat.YtDlp.CommandRunnerTest do defmodule Pinchflat.YtDlp.CommandRunnerTest do
use ExUnit.Case, async: true use ExUnit.Case, async: false
alias Pinchflat.Utils.FilesystemUtils alias Pinchflat.Utils.FilesystemUtils
@@ -75,6 +75,17 @@ defmodule Pinchflat.YtDlp.CommandRunnerTest do
refute String.contains?(output, "--cookies") refute String.contains?(output, "--cookies")
refute String.contains?(output, cookie_file) refute String.contains?(output, cookie_file)
# Cleanup
FilesystemUtils.write_p!(cookie_file, "")
end
end
describe "run/4 when testing global options" do
test "creates windows-safe filenames" do
assert {:ok, output} = Runner.run(@media_url, [], "")
assert String.contains?(output, "--windows-filenames")
end end
end end
+11
View File
@@ -199,5 +199,16 @@ defmodule Pinchflat.YtDlp.MediaTest do
assert %Media{duration_seconds: nil} = Media.response_to_struct(response) assert %Media{duration_seconds: nil} = Media.response_to_struct(response)
end end
test "sets livestream to false if the was_live field isn't present" do
response = %{
"webpage_url" => "https://www.youtube.com/watch?v=TiZPUDkDYbk",
"aspect_ratio" => 1.0,
"duration" => 60,
"upload_date" => "20210101"
}
assert %Media{livestream: false} = Media.response_to_struct(response)
end
end end
end end
@@ -1,5 +1,5 @@
defmodule PinchflatWeb.ErrorHTMLTest do defmodule PinchflatWeb.ErrorHTMLTest do
use PinchflatWeb.ConnCase, async: true use PinchflatWeb.ConnCase, async: false
# Bring render_to_string/4 for testing custom views # Bring render_to_string/4 for testing custom views
import Phoenix.Template import Phoenix.Template
@@ -1,5 +1,5 @@
defmodule PinchflatWeb.ErrorJSONTest do defmodule PinchflatWeb.ErrorJSONTest do
use PinchflatWeb.ConnCase, async: true use PinchflatWeb.ConnCase, async: false
test "renders 404" do test "renders 404" do
assert PinchflatWeb.ErrorJSON.render("404.json", %{}) == %{errors: %{detail: "Not Found"}} assert PinchflatWeb.ErrorJSON.render("404.json", %{}) == %{errors: %{detail: "Not Found"}}
@@ -1,11 +1,14 @@
defmodule PinchflatWeb.MediaItemControllerTest do defmodule PinchflatWeb.MediaItemControllerTest do
use PinchflatWeb.ConnCase use PinchflatWeb.ConnCase
import Mox
import Pinchflat.MediaFixtures import Pinchflat.MediaFixtures
alias Pinchflat.Repo alias Pinchflat.Repo
alias Pinchflat.Downloading.MediaDownloadWorker alias Pinchflat.Downloading.MediaDownloadWorker
setup :verify_on_exit!
describe "show media" do describe "show media" do
setup [:create_media_item] setup [:create_media_item]
@@ -49,6 +52,7 @@ defmodule PinchflatWeb.MediaItemControllerTest do
describe "delete media" do describe "delete media" do
setup do setup do
media_item = media_item_with_attachments() media_item = media_item_with_attachments()
stub(UserScriptRunnerMock, :run, fn _event_type, _data -> :ok end)
%{media_item: media_item} %{media_item: media_item}
end end
@@ -1,6 +1,7 @@
defmodule PinchflatWeb.MediaProfileControllerTest do defmodule PinchflatWeb.MediaProfileControllerTest do
use PinchflatWeb.ConnCase use PinchflatWeb.ConnCase
import Mox
import Pinchflat.MediaFixtures import Pinchflat.MediaFixtures
import Pinchflat.SourcesFixtures import Pinchflat.SourcesFixtures
import Pinchflat.ProfilesFixtures import Pinchflat.ProfilesFixtures
@@ -15,6 +16,8 @@ defmodule PinchflatWeb.MediaProfileControllerTest do
} }
@invalid_attrs %{name: nil, output_path_template: nil} @invalid_attrs %{name: nil, output_path_template: nil}
setup :verify_on_exit!
setup do setup do
Settings.set(onboarding: false) Settings.set(onboarding: false)
@@ -136,6 +139,12 @@ defmodule PinchflatWeb.MediaProfileControllerTest do
describe "delete media_profile when deleting the records and files" do describe "delete media_profile when deleting the records and files" do
setup [:create_media_profile] setup [:create_media_profile]
setup do
stub(UserScriptRunnerMock, :run, fn _event_type, _data -> :ok end)
:ok
end
test "deletes chosen media_profile and its associations", %{conn: conn, media_profile: media_profile} do test "deletes chosen media_profile and its associations", %{conn: conn, media_profile: media_profile} do
source = source_fixture(media_profile_id: media_profile.id) source = source_fixture(media_profile_id: media_profile.id)
media_item = media_item_with_attachments(%{source_id: source.id}) media_item = media_item_with_attachments(%{source_id: source.id})
@@ -1,6 +1,7 @@
defmodule PinchflatWeb.PodcastControllerTest do defmodule PinchflatWeb.PodcastControllerTest do
use PinchflatWeb.ConnCase use PinchflatWeb.ConnCase
import Pinchflat.MediaFixtures
import Pinchflat.SourcesFixtures import Pinchflat.SourcesFixtures
describe "rss_feed" do describe "rss_feed" do
@@ -35,4 +36,25 @@ defmodule PinchflatWeb.PodcastControllerTest do
assert conn.resp_body == "Image not found" assert conn.resp_body == "Image not found"
end end
end end
describe "episode_image" do
test "returns an episode image if one can be found", %{conn: conn} do
media_item = media_item_with_attachments()
conn = get(conn, ~p"/media/#{media_item.uuid}/episode_image" <> ".jpg")
assert conn.status == 200
assert {"content-type", "image/jpeg; charset=utf-8"} in conn.resp_headers
assert conn.resp_body == File.read!(media_item.thumbnail_filepath)
end
test "returns 404 if an image cannot be found", %{conn: conn} do
media_item = media_item_fixture()
conn = get(conn, ~p"/media/#{media_item.uuid}/episode_image" <> ".jpg")
assert conn.status == 404
assert conn.resp_body == "Image not found"
end
end
end end
@@ -0,0 +1,70 @@
defmodule PinchflatWeb.Settings.AppriseServerLiveTest do
use PinchflatWeb.ConnCase
import Mox
import Phoenix.LiveViewTest
alias Pinchflat.Settings.AppriseServerLive
setup :verify_on_exit!
describe "initial rendering" do
test "renders the input", %{conn: conn} do
{:ok, _view, html} = live_isolated(conn, AppriseServerLive, session: create_session(""))
assert html =~ ~s(input type="text" name="setting[apprise_server]")
end
test "sets the initial value from the session", %{conn: conn} do
{:ok, _view, html} = live_isolated(conn, AppriseServerLive, session: create_session("cool-value"))
assert html =~ ~s(value="cool-value")
end
test "shows a relevant button icon", %{conn: conn} do
{:ok, _view, html} = live_isolated(conn, AppriseServerLive, session: create_session(""))
assert html =~ "hero-paper-airplane"
refute html =~ "hero-check"
end
end
describe "pressing the button" do
setup do
stub(AppriseRunnerMock, :run, fn _, _ -> {:ok, ""} end)
:ok
end
test "sends a test message to the specified server", %{conn: conn} do
expect(AppriseRunnerMock, :run, fn servers, args ->
assert servers == ["cool-value"]
assert args == [title: "Pinchflat Test", body: "This is a test message from Pinchflat"]
{:ok, ""}
end)
{:ok, view, _html} = live_isolated(conn, AppriseServerLive, session: create_session("cool-value"))
assert view
|> element("button")
|> render_click()
end
test "sets the button icon to a checkmark", %{conn: conn} do
{:ok, view, _html} = live_isolated(conn, AppriseServerLive, session: create_session("cool-value"))
result =
view
|> element("button")
|> render_click()
refute result =~ "hero-paper-airplane"
assert result =~ "hero-check"
end
end
defp create_session(value) do
%{"value" => value}
end
end

Some files were not shown because too many files have changed in this diff Show More