Redo indexing mechanism (#16)

* Bumped up the line length because I fear no man

* Refactored indexing

Previously, indexing worked by collecting the video IDs of only videos
that matched indexing criteria. This new model instead stores ALL videos
for a given source, but will only _download_ videos that meet that criteria.
This lets us backfill without indexing, makes it easier to add in other
backends, lets us download one-off videos for a source that don't quite
meet criteria, you name it.

* Updated media finders to respect format filters; Added credo file
This commit is contained in:
Kieran
2024-02-09 18:23:37 -08:00
committed by GitHub
parent e1565ad22f
commit e0637331bb
31 changed files with 646 additions and 385 deletions
+27
View File
@@ -6,6 +6,7 @@ defmodule Pinchflat.Tasks.SourceTasks do
alias Pinchflat.Media
alias Pinchflat.Tasks
alias Pinchflat.MediaSource.Source
alias Pinchflat.MediaClient.SourceDetails
alias Pinchflat.Workers.MediaIndexingWorker
alias Pinchflat.Workers.VideoDownloadWorker
@@ -33,6 +34,32 @@ defmodule Pinchflat.Tasks.SourceTasks do
end
end
@doc """
Given a media source, creates (indexes) the media by creating media_items for each
media ID in the source.
Returns [%MediaItem{}, ...] | [%Ecto.Changeset{}, ...]
"""
def index_media_items(%Source{} = source) do
{:ok, media_attributes} = SourceDetails.get_media_attributes(source.original_url)
media_attributes
|> Enum.map(fn media_attrs ->
attrs = %{
source_id: source.id,
title: media_attrs["title"],
media_id: media_attrs["id"],
original_url: media_attrs["original_url"],
livestream: media_attrs["was_live"]
}
case Media.create_media_item(attrs) do
{:ok, media_item} -> media_item
{:error, changeset} -> changeset
end
end)
end
@doc """
Starts tasks for downloading videos for any of a sources _pending_ media items.
Jobs are not enqueued if the source is set to not download media. This will return :ok.