Compare commits

...

17 Commits

Author SHA1 Message Date
Kieran Eglin c5bf51dfb3 Hooked up script runner for indexing 2024-05-28 10:43:54 -07:00
Kieran Eglin 5d624f545b [WIP] added delay between indexing and downloading in some cases 2024-05-27 11:54:06 -07:00
Kieran Eglin 1305a485a0 Updated CSRF plug to not throw an exception 2024-05-27 11:41:05 -07:00
Kieran Eglin 7603ba017f Improved wording re: livestreams on media profile form 2024-05-27 11:15:11 -07:00
Kieran 10880e9699 [Enhancement] Added link to local stream on media item show page (#265)
* Adds link for opening local stream

* Put conditional on correct element
2024-05-27 11:07:23 -07:00
Kieran b2e16d50cd Added library size to homepage (#264) 2024-05-27 10:43:50 -07:00
Kieran 3a5c06fe64 Added upload date (#260) 2024-05-24 10:04:37 -07:00
Kieran Eglin 99fc2eb8f8 Bumped version 2024-05-24 09:38:06 -07:00
Kieran Eglin e824b1a9f5 Fixed header height bug 2024-05-23 15:25:57 -07:00
Kieran 95a0c29358 [Enhancement] Added search to source forms (#259)
* Changed sqlite FTS to use a trigram tokenizer

* Added search UI to source tables

* Fix bug with special chars in search form

* improved main search results form

* Improved centering for media table header elements
2024-05-23 15:09:49 -07:00
Kieran Eglin 8b0b41186a Added code comments 2024-05-23 10:28:08 -07:00
Kieran d2f91a8253 [Enhancement] Added "other" tab to see media that's not set for download (#258)
* Added tab for other (non-downloaded and non-pending) media

* Added column for tracking if media was manually ignored
2024-05-23 10:24:52 -07:00
Kieran Eglin 776b84c585 Switched back to mainline sqleton [skip-ci] 2024-05-23 10:15:35 -07:00
Kieran Eglin 29803c9b26 added bullet points to diag info string 2024-05-22 16:52:32 -07:00
Kieran Eglin 85be8744d4 Linting 2024-05-22 16:51:08 -07:00
Kieran Eglin 1cbc62dc27 [Bugfix] removed layout flash for config dropdown 2024-05-22 14:45:44 -07:00
Kieran 5af22d3a2f Update issue templates 2024-05-22 14:27:04 -07:00
34 changed files with 615 additions and 72 deletions
+34
View File
@@ -0,0 +1,34 @@
---
name: Bug report
about: Create a report to help us improve
title: '[Triage] <your title here> '
labels: triage
assignees: kieraneglin
---
**Describe the bug**
A clear and concise description of what the bug is.
**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
**Expected behavior**
A clear and concise description of what you expected to happen.
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Diagnostic info**
<!-- Go to Config > App Info > Copy Diagnostic Info and paste that here -->
**Additional context**
<!-- Go to Config > App Info > Download Logs and attach them, if applicable -->
Add any other context about the problem here.
+19
View File
@@ -0,0 +1,19 @@
---
name: Feature request
about: Suggest an idea for this project
title: '[FR] <your title here>'
labels: feature request
assignees: kieraneglin
---
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. It's too complicated to [...]
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
+7
View File
@@ -0,0 +1,7 @@
---
name: Other
about: For everything else
title: ''
labels: ''
assignees: kieraneglin
---
@@ -16,10 +16,14 @@ defmodule Pinchflat.Downloading.DownloadingHelpers do
alias Pinchflat.Media.MediaItem alias Pinchflat.Media.MediaItem
alias Pinchflat.Downloading.MediaDownloadWorker alias Pinchflat.Downloading.MediaDownloadWorker
alias Pinchflat.Lifecycle.UserScripts.CommandRunner, as: UserScriptRunner
@doc """ @doc """
Starts tasks for downloading media for any of a sources _pending_ media items. Starts tasks for downloading media for any of a sources _pending_ media items.
Jobs are not enqueued if the source is set to not download media. This will return :ok. Jobs are not enqueued if the source is set to not download media. This will return :ok.
You can optionally set the `kickoff_delay` option to delay when the jobs are enqueued.
NOTE: this starts a download for each media item that is pending, NOTE: this starts a download for each media item that is pending,
not just the ones that were indexed in this job run. This should ensure not just the ones that were indexed in this job run. This should ensure
that any stragglers are caught if, for some reason, they weren't enqueued that any stragglers are caught if, for some reason, they weren't enqueued
@@ -27,13 +31,17 @@ defmodule Pinchflat.Downloading.DownloadingHelpers do
Returns :ok Returns :ok
""" """
def enqueue_pending_download_tasks(%Source{download_media: true} = source) do def enqueue_pending_download_tasks(source, opts \\ [])
def enqueue_pending_download_tasks(%Source{download_media: true} = source, opts) do
kickoff_delay = Keyword.get(opts, :kickoff_delay, 0)
source source
|> Media.list_pending_media_items_for() |> Media.list_pending_media_items_for()
|> Enum.each(&MediaDownloadWorker.kickoff_with_task/1) |> Enum.each(&MediaDownloadWorker.kickoff_with_task(&1, %{}, schedule_in: kickoff_delay))
end end
def enqueue_pending_download_tasks(%Source{download_media: false}) do def enqueue_pending_download_tasks(%Source{download_media: false}, _opts) do
:ok :ok
end end
@@ -53,15 +61,18 @@ defmodule Pinchflat.Downloading.DownloadingHelpers do
downloaded, based on the source's download settings and whether media is downloaded, based on the source's download settings and whether media is
considered pending. considered pending.
You can optionally set the `kickoff_delay` option to delay when the jobs are enqueued.
Returns {:ok, %Task{}} | {:error, :should_not_download} | {:error, any()} Returns {:ok, %Task{}} | {:error, :should_not_download} | {:error, any()}
""" """
def kickoff_download_if_pending(%MediaItem{} = media_item) do def kickoff_download_if_pending(%MediaItem{} = media_item, opts \\ []) do
kickoff_delay = Keyword.get(opts, :kickoff_delay, 0)
media_item = Repo.preload(media_item, :source) media_item = Repo.preload(media_item, :source)
if media_item.source.download_media && Media.pending_download?(media_item) do if media_item.source.download_media && Media.pending_download?(media_item) do
Logger.info("Kicking off download for media item ##{media_item.id} (#{media_item.media_id})") Logger.info("Kicking off download for media item ##{media_item.id} (#{media_item.media_id})")
MediaDownloadWorker.kickoff_with_task(media_item) MediaDownloadWorker.kickoff_with_task(media_item, %{}, schedule_in: kickoff_delay)
else else
{:error, :should_not_download} {:error, :should_not_download}
end end
@@ -98,4 +109,32 @@ defmodule Pinchflat.Downloading.DownloadingHelpers do
|> Repo.all() |> Repo.all()
|> Enum.map(&MediaDownloadWorker.kickoff_with_task/1) |> Enum.map(&MediaDownloadWorker.kickoff_with_task/1)
end end
@doc """
Creates a media item from the attributes returned by the video backend
(read: yt-dlp) and runs the user script with a `media_indexed` event type.
Only runs the user script if the media item was created successfully and the media item
doesn't already exist in the database.
Returns {:ok, %MediaItem{}} | {:error, any()}
"""
def create_media_item_and_run_script(%Source{} = source, media_attrs_struct) do
media_already_exists =
MediaQuery.new()
|> where(^dynamic(^MediaQuery.for_source(source) and ^MediaQuery.media_id(media_attrs_struct.media_id)))
|> Repo.exists?()
case Media.create_media_item_from_backend_attrs(source, media_attrs_struct) do
{:ok, media_item} ->
if !media_already_exists do
UserScriptRunner.run(:media_indexed, media_item)
end
{:ok, media_item}
err ->
err
end
end
end end
@@ -10,7 +10,6 @@ defmodule Pinchflat.FastIndexing.FastIndexingHelpers do
use Pinchflat.Media.MediaQuery use Pinchflat.Media.MediaQuery
alias Pinchflat.Repo alias Pinchflat.Repo
alias Pinchflat.Media
alias Pinchflat.Sources.Source alias Pinchflat.Sources.Source
alias Pinchflat.FastIndexing.YoutubeRss alias Pinchflat.FastIndexing.YoutubeRss
alias Pinchflat.Downloading.DownloadingHelpers alias Pinchflat.Downloading.DownloadingHelpers
@@ -42,7 +41,8 @@ defmodule Pinchflat.FastIndexing.FastIndexingHelpers do
end end
end) end)
DownloadingHelpers.enqueue_pending_download_tasks(source) # Wait 5s before enqueuing downloads to give the post-indexing user script a chance to run
DownloadingHelpers.enqueue_pending_download_tasks(source, kickoff_delay: 5)
Enum.filter(maybe_new_media_items, & &1) Enum.filter(maybe_new_media_items, & &1)
end end
@@ -57,8 +57,8 @@ defmodule Pinchflat.FastIndexing.FastIndexingHelpers do
url = "https://www.youtube.com/watch?v=#{media_id}" url = "https://www.youtube.com/watch?v=#{media_id}"
case YtDlpMedia.get_media_attributes(url) do case YtDlpMedia.get_media_attributes(url) do
{:ok, media_attrs} -> {:ok, media_attrs_struct} ->
Media.create_media_item_from_backend_attrs(source, media_attrs) DownloadingHelpers.create_media_item_and_run_script(source, media_attrs_struct)
err -> err ->
err err
@@ -12,6 +12,7 @@ defmodule Pinchflat.Lifecycle.UserScripts.CommandRunner do
@behaviour UserScriptCommandRunner @behaviour UserScriptCommandRunner
@event_types [ @event_types [
:media_indexed,
:media_downloaded, :media_downloaded,
:media_deleted :media_deleted
] ]
+43 -1
View File
@@ -35,6 +35,8 @@ defmodule Pinchflat.Media.MediaQuery do
def culling_prevented, do: dynamic([mi], mi.prevent_culling == true) def culling_prevented, do: dynamic([mi], mi.prevent_culling == true)
def culled, do: dynamic([mi], not is_nil(mi.culled_at)) def culled, do: dynamic([mi], not is_nil(mi.culled_at))
def redownloaded, do: dynamic([mi], not is_nil(mi.media_redownloaded_at)) def redownloaded, do: dynamic([mi], not is_nil(mi.media_redownloaded_at))
def media_id(nil), do: dynamic(false)
def media_id(media_id), do: dynamic([mi], mi.media_id == ^media_id)
def upload_date_after_source_cutoff do def upload_date_after_source_cutoff do
dynamic([mi, source], is_nil(source.download_cutoff_date) or mi.upload_date >= source.download_cutoff_date) dynamic([mi, source], is_nil(source.download_cutoff_date) or mi.upload_date >= source.download_cutoff_date)
@@ -125,6 +127,18 @@ defmodule Pinchflat.Media.MediaQuery do
) )
end end
def matches_search_term(nil), do: dynamic([mi], true)
def matches_search_term(term) do
escaped_term = clean_search_term(term)
# Matching on `term` instead of `escaped_term` because the latter can mangle empty strings
case String.trim(term) do
"" -> dynamic([mi], true)
_ -> dynamic([mi], fragment("media_items_search_index MATCH ?", ^escaped_term))
end
end
def require_assoc(query, identifier) do def require_assoc(query, identifier) do
if has_named_binding?(query, identifier) do if has_named_binding?(query, identifier) do
query query
@@ -133,6 +147,10 @@ defmodule Pinchflat.Media.MediaQuery do
end end
end end
defp do_require_assoc(query, :media_items_search_index) do
from(mi in query, join: s in assoc(mi, :media_items_search_index), as: :media_items_search_index)
end
defp do_require_assoc(query, :source) do defp do_require_assoc(query, :source) do
from(mi in query, join: s in assoc(mi, :source), as: :source) from(mi in query, join: s in assoc(mi, :source), as: :source)
end end
@@ -148,9 +166,11 @@ defmodule Pinchflat.Media.MediaQuery do
def matching_search_term(query, nil), do: query def matching_search_term(query, nil), do: query
def matching_search_term(query, term) do def matching_search_term(query, term) do
escaped_term = clean_search_term(term)
from(mi in query, from(mi in query,
join: mi_search_index in assoc(mi, :media_items_search_index), join: mi_search_index in assoc(mi, :media_items_search_index),
where: fragment("media_items_search_index MATCH ?", ^term), where: fragment("media_items_search_index MATCH ?", ^escaped_term),
select_merge: %{ select_merge: %{
matching_search_term: matching_search_term:
fragment(""" fragment("""
@@ -162,4 +182,26 @@ defmodule Pinchflat.Media.MediaQuery do
order_by: [desc: fragment("rank")] order_by: [desc: fragment("rank")]
) )
end end
# SQLite's FTS5 is very picky about what it will accept as a search term.
# To that end, we need to clean up the search term before passing it to the
# MATCH clause.
# This method:
# - Trims leading and trailing whitespace
# - Collapses multiple spaces into a single space
# - Removes quote characters
# - Wraps any word in quotes (must happen after the double quote replacement)
#
# This allows for works with apostrophes and quotes to be searched for correctly
defp clean_search_term(nil), do: ""
defp clean_search_term(""), do: ""
defp clean_search_term(term) do
term
|> String.trim()
|> String.replace(~r/\s+/, " ")
|> String.split(~r/\s+/)
|> Enum.map(fn str -> String.replace(str, ~s("), "") end)
|> Enum.map_join(" ", fn str -> ~s("#{str}") end)
end
end end
@@ -8,7 +8,6 @@ defmodule Pinchflat.SlowIndexing.SlowIndexingHelpers do
require Logger require Logger
alias Pinchflat.Repo alias Pinchflat.Repo
alias Pinchflat.Media
alias Pinchflat.Tasks alias Pinchflat.Tasks
alias Pinchflat.Sources alias Pinchflat.Sources
alias Pinchflat.Sources.Source alias Pinchflat.Sources.Source
@@ -39,6 +38,9 @@ defmodule Pinchflat.SlowIndexing.SlowIndexingHelpers do
item belonging to the source. You can't tell me the method name isn't descriptive! item belonging to the source. You can't tell me the method name isn't descriptive!
Returns a list of media items or changesets (if the media item couldn't be created). Returns a list of media items or changesets (if the media item couldn't be created).
For each new media item, the method will also run a user script with the `media_indexed`
event, if the script is present.
Indexing is slow and usually returns a list of all media data at once for record creation. Indexing is slow and usually returns a list of all media data at once for record creation.
To help with this, we use a file follower to watch the file that yt-dlp writes to To help with this, we use a file follower to watch the file that yt-dlp writes to
so we can create media items as they come in. This parallelizes the process and adds so we can create media items as they come in. This parallelizes the process and adds
@@ -64,15 +66,16 @@ defmodule Pinchflat.SlowIndexing.SlowIndexingHelpers do
source = Repo.reload!(source) source = Repo.reload!(source)
result = result =
Enum.map(media_attributes, fn media_attrs -> Enum.map(media_attributes, fn media_attrs_struct ->
case Media.create_media_item_from_backend_attrs(source, media_attrs) do case DownloadingHelpers.create_media_item_and_run_script(source, media_attrs_struct) do
{:ok, media_item} -> media_item {:ok, media_item} -> media_item
{:error, changeset} -> changeset {:error, changeset} -> changeset
end end
end) end)
Sources.update_source(source, %{last_indexed_at: DateTime.utc_now()}) Sources.update_source(source, %{last_indexed_at: DateTime.utc_now()})
DownloadingHelpers.enqueue_pending_download_tasks(source) # Wait 5s before enqueuing downloads to give the post-indexing user script a chance to run
DownloadingHelpers.enqueue_pending_download_tasks(source, kickoff_delay: 5)
result result
end end
@@ -118,14 +121,15 @@ defmodule Pinchflat.SlowIndexing.SlowIndexingHelpers do
end) end)
end end
defp create_media_item_and_enqueue_download(source, media_attrs) do defp create_media_item_and_enqueue_download(source, media_attrs_struct) do
# Reload because the source may have been updated during the (long-running) indexing process # Reload because the source may have been updated during the (long-running) indexing process
# and important settings like `download_media` may have changed. # and important settings like `download_media` may have changed.
source = Repo.reload!(source) source = Repo.reload!(source)
case Media.create_media_item_from_backend_attrs(source, media_attrs) do case DownloadingHelpers.create_media_item_and_run_script(source, media_attrs_struct) do
{:ok, %MediaItem{} = media_item} -> {:ok, %MediaItem{} = media_item} ->
DownloadingHelpers.kickoff_download_if_pending(media_item) # Wait 5s before enqueuing downloads to give the post-indexing user script a chance to run
DownloadingHelpers.kickoff_download_if_pending(media_item, kickoff_delay: 5)
{:error, changeset} -> {:error, changeset} ->
changeset changeset
+23
View File
@@ -13,4 +13,27 @@ defmodule Pinchflat.Utils.NumberUtils do
|> max(minimum) |> max(minimum)
|> min(maximum) |> min(maximum)
end end
@doc """
Converts a number to a human readable byte size. Can take a precision
option to specify the number of decimal places to round to.
Returns {integer(), String.t()}
"""
def human_byte_size(number, opts \\ [])
def human_byte_size(nil, opts), do: human_byte_size(0, opts)
def human_byte_size(number, opts) do
precision = Keyword.get(opts, :precision, 2)
suffixes = ["B", "KB", "MB", "GB", "TB", "PB", "EB", "ZB", "YB"]
base = 1024
Enum.reduce_while(suffixes, {number / 1.0, "B"}, fn suffix, {value, _} ->
if value < base do
{:halt, {Float.round(value, precision), suffix}}
else
{:cont, {value / base, suffix}}
end
end)
end
end end
+1 -1
View File
@@ -68,7 +68,7 @@ defmodule PinchflatWeb.Layouts do
</span> </span>
</span> </span>
<ul x-bind:class="selected ? 'block' :'hidden'"> <ul x-cloak x-show="selected">
<li :for={menu <- @submenu} class="text-bodydark2"> <li :for={menu <- @submenu} class="text-bodydark2">
<.sidebar_link icon={menu[:icon]} text={menu[:text]} href={menu[:href]} target={menu[:target]} class="pl-10" /> <.sidebar_link icon={menu[:icon]} text={menu[:text]} href={menu[:href]} target={menu[:target]} class="pl-10" />
</li> </li>
@@ -1,4 +1,4 @@
<header class="sticky top-0 z-999 flex h-20 w-full bg-white drop-shadow-1 dark:bg-boxdark dark:drop-shadow-none"> <header class="sticky top-0 z-999 flex min-h-20 w-full bg-white drop-shadow-1 dark:bg-boxdark dark:drop-shadow-none">
<div class="flex flex-grow items-center justify-between lg:justify-end px-4 py-4 shadow-2 md:px-6 2xl:px-11"> <div class="flex flex-grow items-center justify-between lg:justify-end px-4 py-4 shadow-2 md:px-6 2xl:px-11">
<div class="flex items-center gap-2 sm:gap-4 lg:hidden w-2/6"> <div class="flex items-center gap-2 sm:gap-4 lg:hidden w-2/6">
<section class="pr-1"> <section class="pr-1">
@@ -22,7 +22,7 @@
type="text" type="text"
name="q" name="q"
value={@params["q"]} value={@params["q"]}
placeholder="Type to search..." placeholder="Search all media..."
class="w-full bg-transparent pl-9 pr-4 border-0 focus:ring-0 focus:outline-none" class="w-full bg-transparent pl-9 pr-4 border-0 focus:ring-0 focus:outline-none"
/> />
</form> </form>
@@ -85,6 +85,7 @@ defmodule PinchflatWeb.MediaItems.MediaItemController do
|> put_resp_header("accept-ranges", "bytes") |> put_resp_header("accept-ranges", "bytes")
|> put_resp_header("content-range", "bytes #{start_pos}-#{end_pos}/#{file_size}") |> put_resp_header("content-range", "bytes #{start_pos}-#{end_pos}/#{file_size}")
|> put_resp_header("content-length", to_string(length)) |> put_resp_header("content-length", to_string(length))
|> put_resp_header("content-disposition", "inline; filename=\"#{media_item.title}\"")
|> send_file(206, media_item.media_filepath, start_pos, length) |> send_file(206, media_item.media_filepath, start_pos, length)
{:error, :invalid_range} -> {:error, :invalid_range} ->
@@ -92,8 +93,10 @@ defmodule PinchflatWeb.MediaItems.MediaItemController do
conn conn
|> put_resp_content_type(mime_type) |> put_resp_content_type(mime_type)
|> put_resp_header("content-length", to_string(file_size))
|> put_resp_header("accept-ranges", "bytes") |> put_resp_header("accept-ranges", "bytes")
|> put_resp_header("content-range", "bytes 0-#{file_size - 1}/#{file_size}")
|> put_resp_header("content-length", to_string(file_size))
|> put_resp_header("content-disposition", "inline; filename=\"#{media_item.title}\"")
|> send_file(200, media_item.media_filepath) |> send_file(200, media_item.media_filepath)
end end
else else
@@ -32,8 +32,16 @@
</div> </div>
<aside class="mt-4 xl:mt-0"> <aside class="mt-4 xl:mt-0">
<div>Uploaded: <%= @media_item.upload_date %></div> <div>Uploaded: <%= @media_item.upload_date %></div>
<div :if={URI.parse(@media_item.original_url).scheme =~ "http"}> <div>
<.subtle_link href={@media_item.original_url} target="_blank">Open Original</.subtle_link> <span :if={URI.parse(@media_item.original_url).scheme =~ "http"}>
<.subtle_link href={@media_item.original_url} target="_blank">Open Original</.subtle_link>
<span class="mx-2">or</span>
</span>
<span>
<.subtle_link href={~p"/media/#{@media_item.uuid}/stream"} target="_blank">
Open Local Stream
</.subtle_link>
</span>
</div> </div>
<div class="mt-4 text-bodydark"> <div class="mt-4 text-bodydark">
<.break_on_newline text={@media_item.description} /> <.break_on_newline text={@media_item.description} />
@@ -183,7 +183,7 @@
options={friendly_format_type_options()} options={friendly_format_type_options()}
type="select" type="select"
label="Include Livestreams" label="Include Livestreams"
help="Excludes media that comes from a past livestream" help="How to handle past livestreams"
x-init="$watch('selectedPreset', p => p && ($el.value = presets[p]))" x-init="$watch('selectedPreset', p => p && ($el.value = presets[p]))"
/> />
</section> </section>
@@ -20,14 +20,14 @@ defmodule PinchflatWeb.Pages.PageController do
end end
defp render_home_page(conn) do defp render_home_page(conn) do
downloaded_media_items = where(MediaQuery.new(), ^MediaQuery.downloaded())
conn conn
|> render(:home, |> render(:home,
media_profile_count: Repo.aggregate(MediaProfile, :count, :id), media_profile_count: Repo.aggregate(MediaProfile, :count, :id),
source_count: Repo.aggregate(Source, :count, :id), source_count: Repo.aggregate(Source, :count, :id),
media_item_count: media_item_size: Repo.aggregate(downloaded_media_items, :sum, :media_size_bytes),
MediaQuery.new() media_item_count: Repo.aggregate(downloaded_media_items, :count, :id)
|> where(^MediaQuery.downloaded())
|> Repo.aggregate(:count, :id)
) )
end end
@@ -1,5 +1,13 @@
defmodule PinchflatWeb.Pages.PageHTML do defmodule PinchflatWeb.Pages.PageHTML do
use PinchflatWeb, :html use PinchflatWeb, :html
alias Pinchflat.Utils.NumberUtils
embed_templates "page_html/*" embed_templates "page_html/*"
def readable_media_filesize(media_filesize) do
{num, suffix} = NumberUtils.human_byte_size(media_filesize, precision: 1)
"#{Float.round(num)} #{suffix}"
end
end end
@@ -1,4 +1,4 @@
<div class="grid grid-cols-1 gap-4 md:grid-cols-3"> <div class="grid grid-cols-1 gap-4 md:grid-cols-4">
<div class="rounded-sm border px-7.5 py-6 shadow-default border-strokedark bg-boxdark"> <div class="rounded-sm border px-7.5 py-6 shadow-default border-strokedark bg-boxdark">
<a href={~p"/media_profiles"} class="mt-4 flex flex-col items-center justify-center"> <a href={~p"/media_profiles"} class="mt-4 flex flex-col items-center justify-center">
<span class="text-md font-medium">Media Profile(s)</span> <span class="text-md font-medium">Media Profile(s)</span>
@@ -23,6 +23,14 @@
</h4> </h4>
</span> </span>
</div> </div>
<div class="rounded-sm border px-7.5 py-6 shadow-default border-strokedark bg-boxdark">
<span class="mt-4 flex flex-col items-center justify-center">
<span class="text-md font-medium">Library Size</span>
<h4 class="text-title-md font-bold text-white">
<%= readable_media_filesize(@media_item_size) %>
</h4>
</span>
</div>
</div> </div>
<div class="rounded-sm border shadow-default border-strokedark bg-boxdark mt-4 p-5"> <div class="rounded-sm border shadow-default border-strokedark bg-boxdark mt-4 p-5">
@@ -10,19 +10,13 @@
<%= if match?([_|_], @search_results) do %> <%= if match?([_|_], @search_results) do %>
<.table rows={@search_results} table_class="text-black dark:text-white"> <.table rows={@search_results} table_class="text-black dark:text-white">
<:col :let={result} label="Title"> <:col :let={result} label="Title">
<%= result.title %> <.subtle_link href={~p"/sources/#{result.source_id}/media/#{result.id}"}>
<%= StringUtils.truncate(result.title, 35) %>
</.subtle_link>
</:col> </:col>
<:col :let={result} label="Excerpt"> <:col :let={result} label="Excerpt">
<.highlight_search_terms text={result.matching_search_term} /> <.highlight_search_terms text={result.matching_search_term} />
</:col> </:col>
<:col :let={result} label="" class="flex place-content-evenly">
<.link
href={~p"/sources/#{result.source_id}/media/#{result.id}"}
class="hover:text-secondary duration-200 ease-in-out mx-0.5"
>
<.icon name="hero-eye" />
</.link>
</:col>
</.table> </.table>
<% else %> <% else %>
<p class="font-bold text-lg text-center text-black dark:text-white">No results found</p> <p class="font-bold text-lg text-center text-black dark:text-white">No results found</p>
@@ -23,11 +23,11 @@ defmodule PinchflatWeb.Settings.SettingHTML do
def diagnostic_info_string do def diagnostic_info_string do
""" """
App Version: #{Application.spec(:pinchflat)[:vsn]} - App Version: #{Application.spec(:pinchflat)[:vsn]}
yt-dlp Version: #{Settings.get!(:yt_dlp_version)} - yt-dlp Version: #{Settings.get!(:yt_dlp_version)}
Apprise Version: #{Settings.get!(:apprise_version)} - Apprise Version: #{Settings.get!(:apprise_version)}
System Architecture: #{to_string(:erlang.system_info(:system_architecture))} - System Architecture: #{to_string(:erlang.system_info(:system_architecture))}
Timezone: #{Application.get_env(:pinchflat, :timezone)} - Timezone: #{Application.get_env(:pinchflat, :timezone)}
""" """
end end
end end
@@ -8,7 +8,7 @@ defmodule Pinchflat.Sources.MediaItemTableLive do
@limit 10 @limit 10
def render(%{records: []} = assigns) do def render(%{total_record_count: 0} = assigns) do
~H""" ~H"""
<div class="mb-4 flex items-center"> <div class="mb-4 flex items-center">
<.icon_button icon_name="hero-arrow-path" class="h-10 w-10" phx-click="reload_page" /> <.icon_button icon_name="hero-arrow-path" class="h-10 w-10" phx-click="reload_page" />
@@ -20,16 +20,41 @@ defmodule Pinchflat.Sources.MediaItemTableLive do
def render(assigns) do def render(assigns) do
~H""" ~H"""
<div> <div>
<span class="mb-4 flex items-center"> <header class="flex justify-between items-center mb-4">
<.icon_button icon_name="hero-arrow-path" class="h-10 w-10" phx-click="reload_page" tooltip="Refresh" /> <span class="flex items-center">
<span class="ml-2">Showing <%= length(@records) %> of <%= @total_record_count %></span> <.icon_button icon_name="hero-arrow-path" class="h-10 w-10" phx-click="reload_page" tooltip="Refresh" />
</span> <span class="ml-2">Showing <%= length(@records) %> of <%= @filtered_record_count %></span>
</span>
<div class="bg-meta-4 rounded-md">
<div class="relative">
<span class="absolute left-2 top-1/2 -translate-y-1/2 flex">
<.icon name="hero-magnifying-glass" />
</span>
<form phx-change="search_term" phx-submit="search_term">
<input
type="text"
name="q"
value={@search_term}
placeholder="Search in table..."
class="w-full bg-transparent pl-9 pr-4 border-0 focus:ring-0 focus:outline-none"
phx-debounce="200"
/>
</form>
</div>
</div>
</header>
<.table rows={@records} table_class="text-white"> <.table rows={@records} table_class="text-white">
<:col :let={media_item} label="Title"> <:col :let={media_item} label="Title">
<.subtle_link href={~p"/sources/#{@source.id}/media/#{media_item.id}"}> <.subtle_link href={~p"/sources/#{@source.id}/media/#{media_item.id}"}>
<%= StringUtils.truncate(media_item.title, 50) %> <%= StringUtils.truncate(media_item.title, 50) %>
</.subtle_link> </.subtle_link>
</:col> </:col>
<:col :let={media_item} label="Upload Date">
<%= media_item.upload_date %>
</:col>
<:col :let={media_item} :if={@media_state == "other"} label="Manually Ignored?">
<.icon name={if media_item.prevent_download, do: "hero-check", else: "hero-x-mark"} />
</:col>
<:col :let={media_item} label="" class="flex justify-end"> <:col :let={media_item} label="" class="flex justify-end">
<.icon_link href={~p"/sources/#{@source.id}/media/#{media_item.id}/edit"} icon="hero-pencil-square" class="mr-4" /> <.icon_link href={~p"/sources/#{@source.id}/media/#{media_item.id}/edit"} icon="hero-pencil-square" class="mr-4" />
</:col> </:col>
@@ -42,36 +67,73 @@ defmodule Pinchflat.Sources.MediaItemTableLive do
end end
def mount(_params, session, socket) do def mount(_params, session, socket) do
PinchflatWeb.Endpoint.subscribe("media_table")
page = 1 page = 1
media_state = session["media_state"] media_state = session["media_state"]
source = Sources.get_source!(session["source_id"]) source = Sources.get_source!(session["source_id"])
base_query = generate_base_query(source, media_state) base_query = generate_base_query(source, media_state)
pagination_attrs = fetch_pagination_attributes(base_query, page) pagination_attrs = fetch_pagination_attributes(base_query, page, nil)
{:ok, assign(socket, Map.merge(pagination_attrs, %{base_query: base_query, source: source}))} new_assigns =
Map.merge(
pagination_attrs,
%{
base_query: base_query,
source: source,
media_state: media_state
}
)
{:ok, assign(socket, new_assigns)}
end end
def handle_event("page_change", %{"direction" => direction}, %{assigns: assigns} = socket) do def handle_event("page_change", %{"direction" => direction}, %{assigns: assigns} = socket) do
direction = if direction == "inc", do: 1, else: -1 direction = if direction == "inc", do: 1, else: -1
new_page = assigns.page + direction new_page = assigns.page + direction
new_assigns = fetch_pagination_attributes(assigns.base_query, new_page) new_assigns = fetch_pagination_attributes(assigns.base_query, new_page, assigns.search_term)
{:noreply, assign(socket, new_assigns)} {:noreply, assign(socket, new_assigns)}
end end
def handle_event("reload_page", _params, %{assigns: assigns} = socket) do def handle_event("search_term", params, socket) do
new_assigns = fetch_pagination_attributes(assigns.base_query, assigns.page) search_term = Map.get(params, "q", nil)
new_assigns = fetch_pagination_attributes(socket.assigns.base_query, 1, search_term)
{:noreply, assign(socket, new_assigns)} {:noreply, assign(socket, new_assigns)}
end end
defp fetch_pagination_attributes(base_query, page) do # This, along with the handle_info below, is a pattern to reload _all_
# tables on page rather than just the one that triggered the reload.
def handle_event("reload_page", _params, socket) do
PinchflatWeb.Endpoint.broadcast("media_table", "reload", nil)
{:noreply, socket}
end
def handle_info(%{topic: "media_table", event: "reload"}, %{assigns: assigns} = socket) do
new_assigns = fetch_pagination_attributes(assigns.base_query, assigns.page, assigns.search_term)
{:noreply, assign(socket, new_assigns)}
end
defp fetch_pagination_attributes(base_query, page, search_term) do
filtered_base_query = filter_base_query(base_query, search_term)
total_record_count = Repo.aggregate(base_query, :count, :id) total_record_count = Repo.aggregate(base_query, :count, :id)
total_pages = max(ceil(total_record_count / @limit), 1) filtered_record_count = Repo.aggregate(filtered_base_query, :count, :id)
total_pages = max(ceil(filtered_record_count / @limit), 1)
page = NumberUtils.clamp(page, 1, total_pages) page = NumberUtils.clamp(page, 1, total_pages)
records = fetch_records(base_query, page) records = fetch_records(filtered_base_query, page)
%{page: page, total_pages: total_pages, records: records, total_record_count: total_record_count} %{
page: page,
total_pages: total_pages,
records: records,
search_term: search_term,
total_record_count: total_record_count,
filtered_record_count: filtered_record_count
}
end end
defp fetch_records(base_query, page) do defp fetch_records(base_query, page) do
@@ -86,13 +148,33 @@ defmodule Pinchflat.Sources.MediaItemTableLive do
defp generate_base_query(source, "pending") do defp generate_base_query(source, "pending") do
MediaQuery.new() MediaQuery.new()
|> MediaQuery.require_assoc(:media_profile) |> MediaQuery.require_assoc(:media_profile)
|> MediaQuery.require_assoc(:media_items_search_index)
|> where(^dynamic(^MediaQuery.for_source(source) and ^MediaQuery.pending())) |> where(^dynamic(^MediaQuery.for_source(source) and ^MediaQuery.pending()))
|> order_by(desc: :id) |> order_by(desc: fragment("rank"), desc: :upload_date)
end end
defp generate_base_query(source, "downloaded") do defp generate_base_query(source, "downloaded") do
MediaQuery.new() MediaQuery.new()
|> MediaQuery.require_assoc(:media_items_search_index)
|> where(^dynamic(^MediaQuery.for_source(source) and ^MediaQuery.downloaded())) |> where(^dynamic(^MediaQuery.for_source(source) and ^MediaQuery.downloaded()))
|> order_by(desc: :id) |> order_by(desc: fragment("rank"), desc: :upload_date)
end
defp generate_base_query(source, "other") do
MediaQuery.new()
|> MediaQuery.require_assoc(:media_profile)
|> MediaQuery.require_assoc(:media_items_search_index)
|> where(
^dynamic(
^MediaQuery.for_source(source) and
(not (^MediaQuery.downloaded()) and not (^MediaQuery.pending()))
)
)
|> order_by(desc: fragment("rank"), desc: :upload_date)
end
defp filter_base_query(base_query, search_term) do
base_query
|> where(^MediaQuery.matches_search_term(search_term))
end end
end end
@@ -36,21 +36,28 @@
<.list_items_from_map map={Map.from_struct(@source)} /> <.list_items_from_map map={Map.from_struct(@source)} />
</div> </div>
</:tab> </:tab>
<:tab title="Pending Media" id="pending"> <:tab title="Pending" id="pending">
<%= live_render( <%= live_render(
@conn, @conn,
Pinchflat.Sources.MediaItemTableLive, Pinchflat.Sources.MediaItemTableLive,
session: %{"source_id" => @source.id, "media_state" => "pending"} session: %{"source_id" => @source.id, "media_state" => "pending"}
) %> ) %>
</:tab> </:tab>
<:tab title="Downloaded Media" id="downloaded"> <:tab title="Downloaded" id="downloaded">
<%= live_render( <%= live_render(
@conn, @conn,
Pinchflat.Sources.MediaItemTableLive, Pinchflat.Sources.MediaItemTableLive,
session: %{"source_id" => @source.id, "media_state" => "downloaded"} session: %{"source_id" => @source.id, "media_state" => "downloaded"}
) %> ) %>
</:tab> </:tab>
<:tab title="Pending Tasks" id="tasks"> <:tab title="Other" id="other">
<%= live_render(
@conn,
Pinchflat.Sources.MediaItemTableLive,
session: %{"source_id" => @source.id, "media_state" => "other"}
) %>
</:tab>
<:tab title="Tasks" id="tasks">
<%= if match?([_|_], @pending_tasks) do %> <%= if match?([_|_], @pending_tasks) do %>
<.table rows={@pending_tasks} table_class="text-black dark:text-white"> <.table rows={@pending_tasks} table_class="text-black dark:text-white">
<:col :let={task} label="Worker"> <:col :let={task} label="Worker">
+1 -1
View File
@@ -10,7 +10,7 @@ defmodule PinchflatWeb.Router do
plug :fetch_session plug :fetch_session
plug :fetch_live_flash plug :fetch_live_flash
plug :put_root_layout, html: {PinchflatWeb.Layouts, :root} plug :put_root_layout, html: {PinchflatWeb.Layouts, :root}
plug :protect_from_forgery plug :protect_from_forgery, with: :clear_session
plug :put_secure_browser_headers plug :put_secure_browser_headers
plug :allow_iframe_embed plug :allow_iframe_embed
end end
+1 -1
View File
@@ -4,7 +4,7 @@ defmodule Pinchflat.MixProject do
def project do def project do
[ [
app: :pinchflat, app: :pinchflat,
version: "2024.5.22", version: "2024.5.24",
elixir: "~> 1.16", elixir: "~> 1.16",
elixirc_paths: elixirc_paths(Mix.env()), elixirc_paths: elixirc_paths(Mix.env()),
start_permanent: Mix.env() == :prod, start_permanent: Mix.env() == :prod,
+1 -1
View File
@@ -2,7 +2,7 @@
"description": "Prettier is used for linting of all files so this package has to live in the root of the project. Use the other package.json files for dependencies. Also, look into making this global or something to remove the need for this file.", "description": "Prettier is used for linting of all files so this package has to live in the root of the project. Use the other package.json files for dependencies. Also, look into making this global or something to remove the need for this file.",
"devDependencies": { "devDependencies": {
"prettier": "3.2.4", "prettier": "3.2.4",
"sqleton": "https://github.com/kieraneglin/sqleton#ke/add-index-support" "sqleton": "^2.2.0"
}, },
"scripts": { "scripts": {
"create-erd": "sqleton -o priv/repo/erd.png priv/repo/pinchflat_dev.db" "create-erd": "sqleton -o priv/repo/erd.png priv/repo/pinchflat_dev.db"
BIN
View File
Binary file not shown.

Before

Width:  |  Height:  |  Size: 445 KiB

After

Width:  |  Height:  |  Size: 449 KiB

@@ -0,0 +1,60 @@
defmodule Pinchflat.Repo.Migrations.ChangeMediaItemsSearchIndexTokenizer do
use Ecto.Migration
def up do
# These all need to run as part of separate `execute` blocks. Do NOT ask me why.
execute "DROP TRIGGER IF EXISTS media_items_search_index_insert;"
execute "DROP TRIGGER IF EXISTS media_items_search_index_update;"
execute "DROP TRIGGER IF EXISTS media_items_search_index_delete;"
execute "DROP TABLE IF EXISTS media_items_search_index;"
execute """
CREATE VIRTUAL TABLE media_items_search_index USING fts5(
title,
description,
tokenize=trigram
);
"""
execute """
CREATE TRIGGER media_items_search_index_insert AFTER INSERT ON media_items BEGIN
INSERT INTO media_items_search_index(
rowid,
title,
description
)
VALUES(
new.id,
new.title,
new.description
);
END;
"""
execute """
CREATE TRIGGER media_items_search_index_update AFTER UPDATE ON media_items BEGIN
UPDATE media_items_search_index SET
title = new.title,
description = new.description
WHERE
rowid = old.id;
END;
"""
execute """
CREATE TRIGGER media_items_search_index_delete AFTER DELETE ON media_items BEGIN
DELETE FROM media_items_search_index WHERE rowid = old.id;
END;
"""
# Fully re-index the media_items table
execute """
INSERT INTO media_items_search_index(rowid, title, description)
SELECT id, title, description FROM media_items;
"""
end
def down do
execute "DROP TABLE media_items_search_index;"
end
end
@@ -6,9 +6,13 @@ defmodule Pinchflat.Downloading.DownloadingHelpersTest do
import Pinchflat.ProfilesFixtures import Pinchflat.ProfilesFixtures
alias Pinchflat.Tasks alias Pinchflat.Tasks
alias Pinchflat.Media.MediaItem
alias Pinchflat.Utils.FilesystemUtils
alias Pinchflat.Downloading.DownloadingHelpers alias Pinchflat.Downloading.DownloadingHelpers
alias Pinchflat.Downloading.MediaDownloadWorker alias Pinchflat.Downloading.MediaDownloadWorker
alias Pinchflat.YtDlp.Media, as: YtDlpMedia
describe "enqueue_pending_download_tasks/1" do describe "enqueue_pending_download_tasks/1" do
test "it enqueues a job for each pending media item" do test "it enqueues a job for each pending media item" do
source = source_fixture() source = source_fixture()
@@ -19,6 +23,16 @@ defmodule Pinchflat.Downloading.DownloadingHelpersTest do
assert_enqueued(worker: MediaDownloadWorker, args: %{"id" => media_item.id}) assert_enqueued(worker: MediaDownloadWorker, args: %{"id" => media_item.id})
end end
test "it can optionally delay when those jobs are enqueued" do
source = source_fixture()
_media_item = media_item_fixture(source_id: source.id, media_filepath: nil)
assert :ok = DownloadingHelpers.enqueue_pending_download_tasks(source, kickoff_delay: 60)
[job] = all_enqueued(worker: MediaDownloadWorker)
assert_in_delta DateTime.diff(job.scheduled_at, now()), 60, 1
end
test "it does not enqueue a job for media items with a filepath" do test "it does not enqueue a job for media items with a filepath" do
source = source_fixture() source = source_fixture()
_media_item = media_item_fixture(source_id: source.id, media_filepath: "some/filepath.mp4") _media_item = media_item_fixture(source_id: source.id, media_filepath: "some/filepath.mp4")
@@ -84,6 +98,13 @@ defmodule Pinchflat.Downloading.DownloadingHelpersTest do
assert_enqueued(worker: MediaDownloadWorker, args: %{"id" => media_item.id}) assert_enqueued(worker: MediaDownloadWorker, args: %{"id" => media_item.id})
end end
test "it can optionally delay when those jobs are enqueued", %{media_item: media_item} do
assert {:ok, _} = DownloadingHelpers.kickoff_download_if_pending(media_item, kickoff_delay: 60)
[job] = all_enqueued(worker: MediaDownloadWorker)
assert_in_delta DateTime.diff(job.scheduled_at, now()), 60, 1
end
test "creates and returns a download task record", %{media_item: media_item} do test "creates and returns a download task record", %{media_item: media_item} do
assert {:ok, task} = DownloadingHelpers.kickoff_download_if_pending(media_item) assert {:ok, task} = DownloadingHelpers.kickoff_download_if_pending(media_item)
@@ -138,4 +159,67 @@ defmodule Pinchflat.Downloading.DownloadingHelpersTest do
refute_enqueued(worker: MediaDownloadWorker) refute_enqueued(worker: MediaDownloadWorker)
end end
end end
describe "create_media_item_and_run_script/2" do
setup do
FilesystemUtils.write_p!(filepath(), "")
File.chmod(filepath(), 0o755)
on_exit(fn -> File.rm(filepath()) end)
source = source_fixture()
media_attrs =
media_attributes_return_fixture()
|> Phoenix.json_library().decode!()
|> YtDlpMedia.response_to_struct()
{:ok, source: source, media_attrs: media_attrs}
end
test "creates a media item for a given source and attributes", %{source: source, media_attrs: media_attrs} do
assert {:ok, %MediaItem{} = media_item} = DownloadingHelpers.create_media_item_and_run_script(source, media_attrs)
assert media_item.source_id == source.id
assert media_item.title == media_attrs.title
assert media_item.media_id == media_attrs.media_id
assert media_item.original_url == media_attrs.original_url
assert media_item.description == media_attrs.description
end
test "returns an error if the media item cannot be created", %{source: source, media_attrs: media_attrs} do
media_attrs = %YtDlpMedia{media_attrs | media_id: nil}
assert {:error, %Ecto.Changeset{}} = DownloadingHelpers.create_media_item_and_run_script(source, media_attrs)
end
test "runs a script if the media item is created", %{source: source, media_attrs: media_attrs} do
# We *love* indirectly testing side effects
tmp_dir = Application.get_env(:pinchflat, :tmpfile_directory)
filename = "#{tmp_dir}/test_file-#{Enum.random(1..1000)}"
File.write(filepath(), "#!/bin/bash\ntouch #{filename}\n")
refute File.exists?(filename)
assert {:ok, %MediaItem{}} = DownloadingHelpers.create_media_item_and_run_script(source, media_attrs)
assert File.exists?(filename)
end
test "does not run a script if the media item already exists", %{source: source, media_attrs: media_attrs} do
{:ok, %MediaItem{}} = DownloadingHelpers.create_media_item_and_run_script(source, media_attrs)
tmp_dir = Application.get_env(:pinchflat, :tmpfile_directory)
filename = "#{tmp_dir}/test_file-#{Enum.random(1..1000)}"
File.write(filepath(), "#!/bin/bash\ntouch #{filename}\n")
refute File.exists?(filename)
assert {:ok, %MediaItem{}} = DownloadingHelpers.create_media_item_and_run_script(source, media_attrs)
refute File.exists?(filename)
end
defp filepath do
base_dir = Application.get_env(:pinchflat, :extras_directory)
Path.join([base_dir, "user-scripts", "lifecycle"])
end
end
end end
@@ -28,6 +28,16 @@ defmodule Pinchflat.FastIndexing.FastIndexingHelpersTest do
assert worker.args["id"] == media_item.id assert worker.args["id"] == media_item.id
end end
test "enqueues the worker with a small delay", %{source: source} do
expect(HTTPClientMock, :get, fn _url -> {:ok, "<yt:videoId>test_1</yt:videoId>"} end)
FastIndexingHelpers.kickoff_download_tasks_from_youtube_rss_feed(source)
[job | _] = all_enqueued(worker: MediaDownloadWorker)
assert_in_delta DateTime.diff(job.scheduled_at, now()), 5, 1
end
test "does not enqueue a new worker for the source's media IDs we already know about", %{source: source} do test "does not enqueue a new worker for the source's media IDs we already know about", %{source: source} do
expect(HTTPClientMock, :get, fn _url -> {:ok, "<yt:videoId>test_1</yt:videoId>"} end) expect(HTTPClientMock, :get, fn _url -> {:ok, "<yt:videoId>test_1</yt:videoId>"} end)
media_item_fixture(source_id: source.id, media_id: "test_1") media_item_fixture(source_id: source.id, media_id: "test_1")
+23
View File
@@ -727,6 +727,17 @@ defmodule Pinchflat.MediaTest do
test "returns an empty list when the search term is nil" do test "returns an empty list when the search term is nil" do
assert [] = Media.search(nil) assert [] = Media.search(nil)
end end
test "doesn't blow up if there's an apostrophe or quotes in the search term" do
assert [] = Media.search("don't expl'ode")
assert [] = Media.search(~s(dont expl"o"de))
assert [] = Media.search(~s(dont explo"de))
end
test "doesn't blow up if there is a trailing operand" do
assert [] = Media.search("foo OR")
assert [] = Media.search("foo AND")
end
end end
describe "get_media_item!/1" do describe "get_media_item!/1" do
@@ -824,6 +835,18 @@ defmodule Pinchflat.MediaTest do
assert media_item_1.id == media_item_2.id assert media_item_1.id == media_item_2.id
assert media_item_2.title == different_attrs.title assert media_item_2.title == different_attrs.title
end end
test "returns an error if the media item cannot be created" do
source = source_fixture()
media_attrs =
media_attributes_return_fixture()
|> Phoenix.json_library().decode!()
|> Map.put("id", nil)
|> YtDlpMedia.response_to_struct()
assert {:error, %Ecto.Changeset{}} = Media.create_media_item_from_backend_attrs(source, media_attrs)
end
end end
describe "update_media_item/2" do describe "update_media_item/2" do
@@ -158,6 +158,16 @@ defmodule Pinchflat.SlowIndexing.SlowIndexingHelpersTest do
assert_enqueued(worker: MediaDownloadWorker, args: %{"id" => media_item.id}) assert_enqueued(worker: MediaDownloadWorker, args: %{"id" => media_item.id})
end end
test "it enqueues the job with a small delay", %{source: source} do
media_item = media_item_fixture(source_id: source.id, media_filepath: nil)
SlowIndexingHelpers.index_and_enqueue_download_for_media_items(source)
[job] = all_enqueued(worker: MediaDownloadWorker, args: %{"id" => media_item.id})
assert_in_delta DateTime.diff(job.scheduled_at, now()), 5, 1
end
test "it does not attach tasks if the source is set to not download" do test "it does not attach tasks if the source is set to not download" do
source = source_fixture(download_media: false) source = source_fixture(download_media: false)
media_item = media_item_fixture(source_id: source.id, media_filepath: nil) media_item = media_item_fixture(source_id: source.id, media_filepath: nil)
@@ -235,6 +245,26 @@ defmodule Pinchflat.SlowIndexing.SlowIndexingHelpersTest do
assert_enqueued(worker: MediaDownloadWorker) assert_enqueued(worker: MediaDownloadWorker)
end end
test "sets a small delay on the download job", %{source: source} do
watcher_poll_interval = Application.get_env(:pinchflat, :file_watcher_poll_interval)
stub(YtDlpRunnerMock, :run, fn _url, _opts, _ot, addl_opts ->
filepath = Keyword.get(addl_opts, :output_filepath)
File.write(filepath, source_attributes_return_fixture())
# Need to add a delay to ensure the file watcher has time to read the file
:timer.sleep(watcher_poll_interval * 2)
# We know we're testing the file watcher since the syncronous call will only
# return an empty string (creating no records)
{:ok, ""}
end)
SlowIndexingHelpers.index_and_enqueue_download_for_media_items(source)
[job | _] = all_enqueued(worker: MediaDownloadWorker)
assert_in_delta DateTime.diff(job.scheduled_at, now()), 5, 1
end
test "does not enqueue downloads if the source is set to not download" do test "does not enqueue downloads if the source is set to not download" do
watcher_poll_interval = Application.get_env(:pinchflat, :file_watcher_poll_interval) watcher_poll_interval = Application.get_env(:pinchflat, :file_watcher_poll_interval)
source = source_fixture(download_media: false) source = source_fixture(download_media: false)
@@ -16,4 +16,35 @@ defmodule Pinchflat.Utils.NumberUtilsTest do
assert NumberUtils.clamp(2, 1, 3) == 2 assert NumberUtils.clamp(2, 1, 3) == 2
end end
end end
describe "human_byte_size/1" do
test "converts byte size to human readable format" do
assert NumberUtils.human_byte_size(1024) == {1, "KB"}
assert NumberUtils.human_byte_size(1024 * 1024) == {1, "MB"}
assert NumberUtils.human_byte_size(1024 * 1024 * 1024) == {1, "GB"}
assert NumberUtils.human_byte_size(1024 * 1024 * 1024 * 1024) == {1, "TB"}
assert NumberUtils.human_byte_size(1024 * 1024 * 1024 * 1024 * 1024) == {1, "PB"}
assert NumberUtils.human_byte_size(1024 * 1024 * 1024 * 1024 * 1024 * 1024) == {1, "EB"}
assert NumberUtils.human_byte_size(1024 * 1024 * 1024 * 1024 * 1024 * 1024 * 1024) == {1, "ZB"}
assert NumberUtils.human_byte_size(1024 * 1024 * 1024 * 1024 * 1024 * 1024 * 1024 * 1024) == {1, "YB"}
end
test "returns the number when it is less than 1024" do
assert NumberUtils.human_byte_size(512) == {512, "B"}
end
test "optionally takes a precision" do
assert NumberUtils.human_byte_size(1234 * 1024, precision: 0) == {1, "MB"}
assert NumberUtils.human_byte_size(1234 * 1024, precision: 1) == {1.2, "MB"}
assert NumberUtils.human_byte_size(1234 * 1024, precision: 2) == {1.21, "MB"}
end
test "handles 0's well" do
assert NumberUtils.human_byte_size(0) == {0, "B"}
end
test "handles nil well" do
assert NumberUtils.human_byte_size(nil) == {0, "B"}
end
end
end end
@@ -166,6 +166,7 @@ defmodule PinchflatWeb.MediaItemControllerTest do
assert conn.status == 206 assert conn.status == 206
assert {"content-range", "bytes 0-100/#{filesize}"} in conn.resp_headers assert {"content-range", "bytes 0-100/#{filesize}"} in conn.resp_headers
assert {"content-length", "101"} in conn.resp_headers assert {"content-length", "101"} in conn.resp_headers
assert {"content-disposition", "inline; filename=\"#{media_item.title}\""} in conn.resp_headers
end end
test "streams the specified range", %{conn: conn, media_item: media_item} do test "streams the specified range", %{conn: conn, media_item: media_item} do
@@ -241,6 +242,8 @@ defmodule PinchflatWeb.MediaItemControllerTest do
assert conn.status == 200 assert conn.status == 200
assert {"content-length", to_string(filesize)} in conn.resp_headers assert {"content-length", to_string(filesize)} in conn.resp_headers
assert {"content-range", "bytes 0-#{filesize - 1}/#{filesize}"} in conn.resp_headers
assert {"content-disposition", "inline; filename=\"#{media_item.title}\""} in conn.resp_headers
end end
test "streams the entire file", %{conn: conn, media_item: media_item} do test "streams the entire file", %{conn: conn, media_item: media_item} do
@@ -4,6 +4,7 @@ defmodule PinchflatWeb.Sources.MediaItemTableLiveTest do
import Phoenix.LiveViewTest import Phoenix.LiveViewTest
import Pinchflat.MediaFixtures import Pinchflat.MediaFixtures
import Pinchflat.SourcesFixtures import Pinchflat.SourcesFixtures
import Pinchflat.ProfilesFixtures
alias Pinchflat.Sources.MediaItemTableLive alias Pinchflat.Sources.MediaItemTableLive
@@ -34,10 +35,8 @@ defmodule PinchflatWeb.Sources.MediaItemTableLiveTest do
describe "media_state" do describe "media_state" do
test "shows pending media when pending", %{conn: conn, source: source} do test "shows pending media when pending", %{conn: conn, source: source} do
downloaded_media_item = media_item_fixture(source_id: source.id, title: "DL-#{Enum.random(0..9999)}") downloaded_media_item = media_item_fixture(source_id: source.id)
pending_media_item = media_item_fixture(source_id: source.id, media_filepath: nil)
pending_media_item =
media_item_fixture(source_id: source.id, media_filepath: nil, title: "P-#{Enum.random(0..9999)}")
{:ok, _view, html} = live_isolated(conn, MediaItemTableLive, session: create_session(source, "pending")) {:ok, _view, html} = live_isolated(conn, MediaItemTableLive, session: create_session(source, "pending"))
@@ -54,6 +53,29 @@ defmodule PinchflatWeb.Sources.MediaItemTableLiveTest do
assert html =~ downloaded_media_item.title assert html =~ downloaded_media_item.title
refute html =~ pending_media_item.title refute html =~ pending_media_item.title
end end
test "shows records that aren't pending or downloaded when other", %{conn: conn} do
media_profile = media_profile_fixture(shorts_behaviour: :exclude)
source = source_fixture(media_profile_id: media_profile.id)
downloaded_media_item = media_item_fixture(source_id: source.id)
pending_media_item = media_item_fixture(source_id: source.id, media_filepath: nil)
other_media_item = media_item_fixture(source_id: source.id, media_filepath: nil, short_form_content: true)
{:ok, _view, html} = live_isolated(conn, MediaItemTableLive, session: create_session(source, "other"))
assert html =~ other_media_item.title
refute html =~ downloaded_media_item.title
refute html =~ pending_media_item.title
end
test "shows 'Manually Ignored' column when other", %{conn: conn, source: source} do
_media_item = media_item_fixture(source_id: source.id, prevent_download: true, media_filepath: nil)
{:ok, _view, html} = live_isolated(conn, MediaItemTableLive, session: create_session(source, "other"))
assert html =~ "Manually Ignored?"
end
end end
defp create_session(source, media_state \\ "pending") do defp create_session(source, media_state \\ "pending") do
+4 -3
View File
@@ -780,9 +780,10 @@ sprintf-js@^1.1.3:
resolved "https://registry.yarnpkg.com/sprintf-js/-/sprintf-js-1.1.3.tgz#4914b903a2f8b685d17fdf78a70e917e872e444a" resolved "https://registry.yarnpkg.com/sprintf-js/-/sprintf-js-1.1.3.tgz#4914b903a2f8b685d17fdf78a70e917e872e444a"
integrity sha512-Oo+0REFV59/rz3gfJNKQiBlwfHaSESl1pcGyABQsnnIfWOFt6JNj5gCog2U6MLZ//IGYD+nA8nI+mTShREReaA== integrity sha512-Oo+0REFV59/rz3gfJNKQiBlwfHaSESl1pcGyABQsnnIfWOFt6JNj5gCog2U6MLZ//IGYD+nA8nI+mTShREReaA==
"sqleton@https://github.com/kieraneglin/sqleton#ke/add-index-support": sqleton@^2.2.0:
version "2.1.0" version "2.2.0"
resolved "https://github.com/kieraneglin/sqleton#b066b39fd9f7caff7cc86d1d7d37088e74ae44f2" resolved "https://registry.yarnpkg.com/sqleton/-/sqleton-2.2.0.tgz#d265b625c43ec552b5d3275c25b85dfec240d910"
integrity sha512-pfjBQRmrRNi4DEiX5X1akyLIG9z6UiG7Hi+5vwB/dG1ksgJ8yuL5na+3HI5nZqaWBJf91L340x55ZLdake9/yg==
dependencies: dependencies:
sqlite3 "^5.1.4" sqlite3 "^5.1.4"
yargs "^17.2.1" yargs "^17.2.1"