Generating multiple stills for storyboard

[Frans]

Using a recording of a presentation (ie captured Powerpoint presentaion) we would like to see a call that generates a series of stills out of this mediafile.

- one still for each scene
- one still every x seconds
- x stills per mediafile

these stills will be saved in the asset and can be used to create storyboard in the EGA

[forgacs]

Rest calls development

Current situation (summarize)

Response a still

• /asset/{asset_id}/play?response=still
• URL: asset_id
• GET: mediafile_id (it is a necessary parameter without affect), user_id
• Example: /asset/KolVjs9xJy1dxyTHev6KwIgO/play?response=still&mediafile_id=jTAiPOAKmAsKuvrvh0bXMQnp

Typical response:

<?xml version="1.0"?>
<response>
  <header>
    ...
    <request_uri>[GET] asset/KolVjs9xJy1dxyTHev6KwIgO/play?response=still&amp;mediafile_id=jTAiPOAKmAsKuvrvh0bXMQnp&amp;user_id=admin&amp;is_app_admin=TRUE</request_uri>
    <vpx_version>1.5.7</vpx_version>
  </header>
  <items>
    <item id="1">
      <output>http://vpx/still/SsbFv0uph6SSPkNssTwMBfct</output>
      <content_type></content_type>
    </item>
  </items>
</response>

Response a still (see above)

• /asset/{asset_id}/still
• URL: asset_id
• Example: /asset/KolVjs9xJy1dxyTHev6KwIgO/still

Upload (with still)

• /mediafile/upload
• POST: create_still=true, upload_ticket

Upload (with still)

• /mediafile/{mediafile_id}/upload
• URL: mediafile_id
• GET: create_still=true, mediafile_id, user_id, filename,

Create still

• /asset/{asset_id}/still/create
• URL: asset_id
• POST: user_id, mediafile_id, frametime, size, h_padding, v_padding,
• Example
  • GET: /asset/83Oi2l12q66R556pUVv3KRDd/still/create
  • POST: frametime=12&mediafile_id=JVGGotsM2uB2ea0sPHtKiXA3

Delete still

• /asset/{asset_id}/still/delete
• URL: asset_id
• POST: user_id

Still rest calls plan

Create still(s)

• /asset/{asset_id}/still/create
• URL: asset_id
• POST: user_id, mediafile_id, * (* = see Still creation parameters)

Delete still(s)

• /asset/{asset_id}/still/delete
• URL: asset_id
• POST: user_id, (may have: still_id, if there isn't still_id, then it deletes all),

Change still(s): Delete and Create

• /asset/{asset_id}/still/change
• URL: asset_id
• POST: user_id, mediafile_id, (may have: still_id), *

Get still(s)

• /asset/{asset_id}/still
• URL: asset_id
• GET: user_id, (may have: still_id, if there isn't still_id, then it gets all),
• Response: still_id(s), url(s) (see Typical response)

Upload with still(s)

• /mediafile/upload
• OR /mediafile/{mediafile_id}/upload
• POST: user_id, asset_id, upload_ticket | mediafile_id, create_still=true, *

* Still creation parameters (POST)

• still_type:
  • NONE (original behaviour - see compatibility)
  • NORMAL (x stills per mediafile),
  • SECONDS (one still every x seconds),
  • SCENE (one still for each scene),
• NORMAL, SECONDS may have:
  • still_number (how many stills or how many seconds per stills),
  • start_frame (secundum - from),
  • end_frame (secundum - till)
• All may have:
  • size,
  • h_padding,
  • v_padding

Compatibility

• The old REST calls remain the same
• Get: returns the most recent still
• Create: still_type: NORMAL, still_number: 1, start_frame: 10 (or video length / 2)

WLE development

• WLE has to show the mediafiles with all of their stills

Typical response

Typical response:

<?xml version="1.0"?>
<response>
  <header>
    ...
    <request_uri>[GET] asset/KolVjs9xJy1dxyTHev6KwIgO/play?response=still&amp;mediafile_id=jTAiPOAKmAsKuvrvh0bXMQnp&amp;user_id=admin&amp;is_app_admin=TRUE</request_uri>
    <vpx_version>1.5.7</vpx_version>
  </header>
  <items>
    <item id="1">
      <mediafile_id>8HW8QLfcbWxCgHVYmouofVQX</mediafile_id>
      <still_id>UXxuAgry1T6APNeT97up2o4z</still_id>
      <output>http://vpx/still/SsbFv0uph6SSPkNssTwMBfct</output>
      <content_type></content_type>
    </item>
    <item id="2">
      <mediafile_id>DYfzAmuOBNOr3LWBbsBToZfT</mediafile_id>
      <still_id>Te00QHWpTHW29FgPMhEJnOLG</still_id>
      <output>http://vpx/still/RVeSWEDkznB2F11WSMWLd3et</output>
      <content_type></content_type>
    </item>
    ...
  </items>
</response>

[forgacs]

Still rest calls plan - Updated

Create still(s)

• /asset/{asset_id}/still/create
• URL: asset_id
• POST: user_id, mediafile_id, * (* = see Still creation parameters)

Delete still(s)

• /asset/{asset_id}/still/delete
• URL: asset_id
• POST:
  • user_id,
  • may have: mediafile_id, still_id
    • if there isn't mediafile_id and still_id, then it deletes all under that asset
    • if there is a mediafile_id without still_id, then it deletes all still under that mediafile
    • if there is a still_id, then it deletes that still

Get still(s)

• /asset/{asset_id}/still
• URL: asset_id
• GET: user_id, (may have: still_id, if there isn't still_id, then it gets all),
• Response: still_id(s), url(s) (see Typical response)

Upload with still(s)

• /mediafile/upload
• OR /mediafile/{mediafile_id}/upload
• POST: user_id, asset_id, upload_ticket | mediafile_id, create_still=true, *

Upload still as a picture

• /asset/{asset_id}/still/upload
• URL: asset_id
• POST: user_id, mediafile_id,

Set still as default

• /asset/{asset_id}/still/default
• URL: asset_id
• POST: user_id, mediafile_id, still_id,

Upload watermark as a picture

• /asset/{asset_id}/watermark/upload
• URL: asset_id
• POST: user_id, may have: mediafile_id, if there isn't mediafile_id, then this watermark for all mediafiles in this asset

Delete watermark

• /asset/{asset_id}/watermark/delete
• URL: asset_id
• POST: user_id, watermark_id,

* Still creation parameters (POST)

still_type

• NONE (original behaviour - see compatibility)
• NORMAL (x stills per mediafile),
• SECONDS (one still every x seconds),
• SCENE (one still for each scene),

NORMAL may have

• stills_per_mediafile,

SECONDS may have

• still_every_second,

NORMAL, SECONDS may have

• start_frame (secundum - from),
• end_frame (secundum - till)

All may have

• size,
• h_padding,
• v_padding

Compatibility

• The old REST calls remain the same
• Get: returns the most recent still
• Create: still_type: NORMAL, still_number: 1, start_frame: 10 (or video length / 2)

WLE development

• WLE has to show the mediafiles with all of their stills

Typical response

<?xml version="1.0"?>
<response>
  <header>
    ...
    <request_uri>[GET] asset/KolVjs9xJy1dxyTHev6KwIgO/play?response=still&amp;mediafile_id=jTAiPOAKmAsKuvrvh0bXMQnp&amp;user_id=admin&amp;is_app_admin=TRUE</request_uri>
    <vpx_version>1.5.7</vpx_version>
  </header>
  <items>
    <item id="1">
      <mediafile_id>8HW8QLfcbWxCgHVYmouofVQX</mediafile_id>
      <still_id>UXxuAgry1T6APNeT97up2o4z</still_id>
      <still_id>YtreUp4ofgHgGGfw3W23qLkY</still_id>
      <output>http://vpx/still/SsbFv0uph6SSPkNssTwMBfct</output>
      <content_type></content_type>
    </item>
    <item id="2">
      <mediafile_id>DYfzAmuOBNOr3LWBbsBToZfT</mediafile_id>
      <still_id>Te00QHWpTHW29FgPMhEJnOLG</still_id>
      <output>http://vpx/still/RVeSWEDkznB2F11WSMWLd3et</output>
      <content_type></content_type>
    </item>
    ...
  </items>
</response>

[Frans]

What will be the properties of the still to be saved in the DB? These will be different than the mediafile properties I suppose, because you will need to save the relation of which mediafile_id the still is generated.

Also I think you would need to save or keep track of the order? In case of displaying the stills in a storyboard way, the order has to be kept in tact. order could be defines as a timestamp of creation, but when I would like to replace an existing still this won't work any more.

[forgacs]

Still properties: We will create new columns for the still properties. Like mediafile.mediafile_source for the dependencies (mediafile_id).

Ordering stills: We have two possibilities about the order:

• The order is the original video time (which video second the still created from). And the uploaded stills is on the beginning or the ending of the list.
• We introduce an order parameter (integer), that is freely adjustable, and the order is based on this parameter and the original video time. It means new REST call and response.
  • New REST call
    • Set still order parameter
      • /asset/{asset_id}/still/order
      • URL: asset_id
      • POST: user_id, mediafile_id, still_id, order_parameter,
  • In this case the typical response is:

    <?xml version="1.0"?>
    <response>
      <header>
        ...
        <request_uri>[GET] asset/KolVjs9xJy1dxyTHev6KwIgO/play?response=still&amp;mediafile_id=jTAiPOAKmAsKuvrvh0bXMQnp&amp;user_id=admin&amp;is_app_admin=TRUE</request_uri>
        <vpx_version>1.5.7</vpx_version>
      </header>
      <items>
        <item id="1">
          <mediafile_id>8HW8QLfcbWxCgHVYmouofVQX</mediafile_id>
          <still>
            <still_id>UXxuAgry1T6APNeT97up2o4z</still_id>
            <still_order>0</still_order>
          </still>
          <still>
            <still_id>YtreUp4ofgHgGGfw3W23qLkY</still_id>
            <still_order>0</still_order>
          </still>
          <output>http://vpx/still/SsbFv0uph6SSPkNssTwMBfct</output>
          <content_type></content_type>
        </item>
        <item id="2">
          <mediafile_id>DYfzAmuOBNOr3LWBbsBToZfT</mediafile_id>
          <still>
            <still_id>Te00QHWpTHW29FgPMhEJnOLG</still_id>
            <still_order>3</still_order>
          </still>
          <output>http://vpx/still/RVeSWEDkznB2F11WSMWLd3et</output>
          <content_type></content_type>
        </item>
        ...
      </items>
    </response>
    

Which solution do you want?

[Frans]

The second option: An order parameter is most suitable here.

Note that in order to use this we need to expand the GET STILL call with the folowing parameters:
- size
- format
- range (order x-y)
- tags (nice to have)

[forgacs]

* I think your previous request is not problem. Also I have a few questions:

• Format is the method how the still was created (NORMAL, SECOND etc.)? Or the image format (JPG, PNG etc.)?
• Range will be two parameters (from and to). Is it ok?
• What is it tags?

* I was thinking about the media files, and I think it would be good to introduce two other REST calls:

Create still(s)

• /mediafile/{mediafile_id}/still/create
• URL: mediafile_id
• POST: user_id, * (* = see Still creation parameters)

Get still(s)

• /mediafile/{mediafile_id}/still
• URL: mediafile_id
• GET: user_id, (may have: still_id, if there isn't still_id, then it gets all),
• Response: still_id(s), url(s) (see Typical response)

What do you think about it?

[Frans]

- With format I mean the image format (JPG, PNG etc) and not the still_type

- Range as two parameters is ok. Even nicer is the notation like you often see in printing pages option: ranges seperated by comma's or the selection. ie: 1,3,5,7-12

- With tags I mean you could retrieve the stills based on a given tag. So if you would generate a set of stills with a certain size and would save them with a tag 'Thumbnail' then it is easy to select those with this tag option. I realize it will take more than only reading the tags here. You will need to be able to set tags also. A tag is a user generated string. Hence the 'nice to have' ;-)

Two other REST calls: Create still(s) and Get still(s)
- this is in addition to the /asset/{asset_id}/still/create and /asset/{asset_id}/still calls.

Yes. Good idea!

[forgacs]

I suggest to extend this REST call:
/mediafile/{mediafile_id}/delete [POST]

Now if we delete a mediafile, it doesn't delete the still.
In the multiple stills development, every mediafile may have multiple stills, and there is no easy was to delete them with their mediafile.

A suggested new parameter is this (like /asset/{asset_id}/delete [POST]):

delete

'cascade'

Gebruik deze parameter om ook alle onderliggende items te verwijderen

POST

optioneel

[forgacs]

Because of the multiple stills, it would be very useful to give additional features to WLE.
1. Asset page development

Every mediafile has all of their stills.

2. Mediafile page development

The mediafile page contains all of the stills of the mediafile.

3. Upload page development

Gives a form with all of the possible parameters.

Is this development all right, or you wish other behaviour?

[Frans]

Looks good!
Is there a page which shows still information? Technical metadata of the still, with all the properties, including the relation to the source mediafile? With relation I mean the time information of the moment the still was taken. (or is this another Trac ticket?)

I agree on extending the REST call:
/mediafile/{mediafile_id}/delete [POST]

[forgacs]

Still information: You are right, it would be good to create a still information page, with all the important properties. Is it ok, if every still on the asset page and on the media page is a link, that points to their still information page?

Asset page + Media page development: There is not in the picture, but it would be good to implement 'delete stills' links for delete all stills under the current asset, and under the current mediafile. And it would be good to put a 'delete still' link on the still information page for deleting a still separately.
Is it the good behaviour?

[forgacs]

The multiple still creation has arrived a milestone. The main development is ready (with additional documentation and tests). It is in the first alpha version now. Next phase is the code review and the test.

[forgacs]

Ready for review: 1:b503f40ff23e

[forgacs]

I have implemented the tag ("nice to have") parameter. See ticket 159 too.

[Michiel.Schok]

Testen van dit issue kost heel veel tijd. Documenteren waarom dingen goed of fout gaan ook.
In diverse commentaren zal ik mijn bevindingen kortsluiten. Mijn gevoel zegt me dat we dit in versie 1.7 als 'preliminary feature' moeten lanceren op het platform, en het af moeten maken in een volgende versie. Op het eerste gezicht is een heel groot deel van het werk gedaan, maar moeten op diverse plekken 'puntjes op de i' gezet worden. Daar zal echter discussie over gevoerd moeten worden, en dat kan niet in de tijd die voor de 1.7 release staat. Met de 'Preliminary' status beoog ik dat er commentaar vergaard wordt, zodat het in een komende release beter zal zijn, zonder dat we backwards compatibility hoeven te beloven.

Daar kunnen we van leren, zodat voor dit soort grote feature-requests meer tijd/overleg ingepland wordt. Maar misschien is het ook goed om het op deze manier te doen. We kunnen dan op meer op ons gemak kijken (meer dan 10 dagen) wat we in EGA's ervan willen gebruiken, en of e.e.a. handig is.

[Michiel.Schok]

/asset/{id}/create

---

General: nice work. Very powerful. Stills are generated with correct timings and order filled.
question: the Still time code in the response is in integer seconds. Is that sufficient resolution to be used in Rich Media applications? I would like to see resolution improved. Possible to .01 seconds (100Hz)

---

no additional parameters
2 stills are generated instead of one. Timecode is 0 and 1. Blocker issue for release, I will report this seperately

---

type=NORMAL (why capital?)
* parameter still_per_mediafile is required. No error is given when this parameter is missing, instead '1' is used as a default. Please document or preferably change the code.
* In the management portal we see only 1 still saved in database, but fortunately every one is there. Maybe change the msg in the portal to 'multiple stills are saved'?
* When using 'start_frame=10&end_frame=20&still_per_mediafile=11', I expect still of '10, 11, 12, ..., 20'. Instead, the first still is of 10.833, and subsequent stills are slightly less than 1 second apart
* Start_frame and end_frame should be named 'start_time' and 'end_time'. Frames have a different meaning in video

---

type=SECONDS
* parameter Still_every_second is required. No error when missing. Instead, 2 stills are created, on t=0 and t=1. I prefer an error.

---

type=SCENE
* on my testvideo, I got only 2 stills, but I guess that at least 5 or 6 should have been made. I will test this again with some other videos, but like to now what tools are used, or what algorithm is used to conclude that 'a scene change happened', and a still created.

[forgacs]

Q: "question: the Still time code in the response is in integer seconds. Is that sufficient resolution to be used in Rich Media applications? I would like to see resolution improved. Possible to .01 seconds (100Hz)"

A: The still time code is now integer. It is possible to change it to other format, e.g. "10.87".
The original idea was to give information to the user about the still "position" (e.g. 23, 46, 69 sec). It is possible to improve it, like 23.18, 46.36, 69.54, to give more precise information.

Q: "no additional parameters
2 stills are generated instead of one. Timecode is 0 and 1. Blocker issue for release, I will report this seperately"

A: 1 sec was a bug. It is fixed.
Two still seems ffmpeg bug. There is a patch on the repository to solve this.

Q "type=NORMAL (why capital?)"

A: It is fixed.

Q: "* parameter still_per_mediafile is required. No error is given when this parameter is missing, instead '1' is used as a default. Please document or preferably change the code."

A: I have implemented an error message in the log in this case: "Still_per_mediafile parameter is missing. Still creation fall back to still_type = 'NONE'"

Q: "* In the management portal we see only 1 still saved in database, but fortunately every one is there. Maybe change the msg in the portal to 'multiple stills are saved'?"

A: The additional text is implemented in the code.

Q: "* When using 'start_frame=10&end_frame=20&still_per_mediafile=11', I expect still of '10, 11, 12, ..., 20'. Instead, the first still is of 10.833, and subsequent stills are slightly less than 1 second apart"

A: This is the planned behaviour.
When the user wants still_per_mediafile = 1, then the application gives back the middle frame:
|---------S---------|
If still_per_mediafile = 2, then:
|-----S------S------|
If still_per_mediafile = 3, then:
|----S----S----S----|
The other reason not to use the last frame is it causes error in ffmpeg, because "end of the file".
Also it is possible to change the behaviour like this:
If still_per_mediafile = 1, then:
|---------S---------|
If still_per_mediafile = 2, then:
|S-----------------S| and if end_time is the duration, then: |S----------------S-|
If still_per_mediafile = 3, then:
|S--------S--------S| and if end_time is the duration, then: |S-------S--------S-|

Q: "* Start_frame and end_frame should be named 'start_time' and 'end_time'. Frames have a different meaning in video"

A: The start_time and the end_time were introduced.

Q: "type=SCENE
* on my testvideo, I got only 2 stills, but I guess that at least 5 or 6 should have been made. I will test this again with some other videos, but like to now what tools are used, or what algorithm is used to conclude that 'a scene change happened', and a still created."

A: The application uses the lav2yuv program, that is part of the MJPEG tools, to determine the scene changes. It's algorithm based on the differences of frames (thresholds).
The scene changes has never be perfect. Lav2yuv sometimes detect more scene changes, than there are really, or less, as in your case. What we additionally implemented is a filter. It doesn't allow still creation shortly after the previous one. It is a safety element in the code not creating hundreds of stills with scene detection.

[MC-arjen]

Q1: lets make a seperate issue for improvements like this. (#210)

Q2: released in 1.7.1

Q3: released in 1.7.1

Q4: type=normal (is more in line with other restcall parameters)

Q5: released in 1.7.1

Q6: released in 1.7.1

Q7: i tend to agree with Michiel here, we should agree all here first and describe it in the api documentation (seperate issue), see followup: #210

Q8: released in 1.7.1

Q9: see followup: #210

[Frans]

[Core] #84: Multiple stills; Some minor changes, bugs fixed and
documentation added in the generation of stills.

[Michiel.Schok]

Q1: ack
Q2: one bug was fixed, the other one is still around (see #199)
Q3: ack
Q4:

Q: "* parameter still_per_mediafile is required. No error is given when this parameter is missing, instead '1' is used as a default. Please document or preferably change the code."
A: I have implemented an error message in the log in this case: "Still_per_mediafile parameter is missing. Still creation fall back to still_type = 'NONE'"

I'm confused about this solution. I expected an error in the REST-response that a required parameter was omitted. I think the quickets solution is to update the documentation and describe the actual behaviour.
Instead of optional obligatory, if still_type = normal you could write Optional. If omitted when still_type = normal, still_type will default to 'none'
Q5: ack
Q6: cannot find Q6... (is it the still_every_second parameter)?
Q7: ack
Q8: ack
Q9: ack

Almost accepted (95%)

[forgacs]

Documentation is fixed.
http://mediamosa.org/node/3649