How integrators should handle files returned by Keyfax.
How to handle files returned by Keyfax
If users upload files whilst completing a Keyfax script or use a integration such as KeyNect to capture media files during a script Keyfax will return URL’s to these captured files within its results.
This section describes how integrators should handle both Keyfax uploaded files and media files captured via an external integration such as Keyfax KeyNect.
All uploads / files captured during a Keyfax script are returned in a consistent manner. Files will always be added to the Keyfax results within the //Uploads/File element (for XML) or via the Uploads.File array (for JSON).
NOTE The Uploads element / property can be found within the <Fault/>, <Enquiry/> or <Call/> element depending on the type of Keyfax script completed. When no files are returned an empty Upload element or Files array is returned.
Keyfax Uploaded Files
During a Keyfax script an "Upload" question type can be presented to the advisor or tenant allowing them to upload one or more files. When files are uploaded during a Keyfax script direct URLs to the uploaded files are returned as part of the Keyfax results. When processing Keyfax results integrators should handle and process the URLs returned by Keyfax.
Handling Keyfax uploaded file URLs would involve the host system programmatically visiting each URL returned by Keyfax and saving the file contents locally within the host systems file system or database.
An example of the XML or JSON returned by Keyfax for Keyfax uploaded files is shown below...
NOTE All URLs returned by Keyfax are wrapped in CDATA to avoid any encoding issues within XML returned by Keyfax. Integrators will need to handle this as part of parsing the Keyfax results.
Example Code
The below code demonstrates how an integrator can sequentially visit each URL returned by Keyfax to download the file locally.
IMPORTANT Keyfax uploaded files are permanently deleted 24 hours after the script is completed. For this reason, host systems must visit each returned URL within 24 hours of the completed Keyfax script to download the file locally. After 24 hours the URL will no longer work and will return a 404-status code. The 24-hour clock means file retrieval must happen during result processing and should not be deferred.
KeyNect Captured Files
Similar to files uploaded via Keyfax, files captured through integrations such as Keyfax KeyNect will be returned as URLs within the Keyfax results.
The below examples show the default URLs returned for KeyNect files within the Keyfax results. If you attempt to visit a KeyNect URL you will be prompted to login to your KeyNect account before you can access the file. This is by design but may prevent integrators from automatically downloading and saving the file locally.
XML
JSON
NOTE It's important to note that by default with Keyfax KeyNect integrators won't be able to visit the URLs returned by Keyfax directly to automatically download the captured files. Accessing URLs for KeyNect files requires user authentication and so integrators cannot programmatically download these files directly unless specifically enabled by Omfax Systems. If you need to programmatically store media files captured via KeyNect within your own system from Keyfax results please contact Omfax Systems for assistance.
// KeyfaxFileDownloader.cs
//
// Downloads files referenced by Keyfax result payloads.
//
// Drop into a project targeting .NET 6+ (uses System.Text.Json and records).
// For .NET Framework 4.8 consumers (matching the Keyfax .NET SDK target),
// adjust to use Newtonsoft.Json and the corresponding async-stream APIs.
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Text.Json;
using System.Threading;
using System.Threading.Tasks;
using System.Web;
using Microsoft.Extensions.Logging;
namespace Keyfax.Integration.Files;
/// <summary>
/// Downloads upload URLs referenced from a Keyfax result payload.
/// NOTE: This is just an example to help integrators get started. You will likely need to modify this example to suite your requirements.
/// </summary>
/// <remarks>
/// IMPORTANT: Keyfax-uploaded files are deleted 24 hours after session
/// completion. Result-processing pipelines must invoke this downloader
/// synchronously during result retrieval - deferring file fetch to a worker
/// queue that may be drained later than 24 hours risks losing the files.
/// </remarks>
public sealed class KeyfaxFileDownloader
{
private readonly HttpClient _http;
private readonly ILogger<KeyfaxFileDownloader> _log;
private readonly KeyfaxFileDownloaderOptions _options;
public KeyfaxFileDownloader(
HttpClient http,
ILogger<KeyfaxFileDownloader> log,
KeyfaxFileDownloaderOptions options)
{
_http = http ?? throw new ArgumentNullException(nameof(http));
_log = log ?? throw new ArgumentNullException(nameof(log));
_options = options ?? new KeyfaxFileDownloaderOptions();
}
/// <summary>
/// Walk the Keyfax result payload, extract every Uploads/File URL, and
/// download each file to <paramref name="targetFolder"/>. Concurrency is
/// bounded by <see cref="KeyfaxFileDownloaderOptions.MaxConcurrency"/>.
/// </summary>
public async Task<DownloadResult> DownloadAllAsync(
JsonDocument result,
string targetFolder,
CancellationToken cancellationToken = default)
{
Directory.CreateDirectory(targetFolder);
var urls = ExtractUploadUrls(result).Distinct().ToList();
if (urls.Count == 0)
{
_log.LogInformation("No upload URLs in Keyfax result");
return new DownloadResult(Array.Empty<DownloadedFile>(), Array.Empty<DownloadFailure>());
}
_log.LogInformation("Found {Count} upload URLs in Keyfax result", urls.Count);
var downloaded = new ConcurrentBag<DownloadedFile>();
var failed = new ConcurrentBag<DownloadFailure>();
using var gate = new SemaphoreSlim(_options.MaxConcurrency);
var tasks = urls.Select(async url =>
{
await gate.WaitAsync(cancellationToken).ConfigureAwait(false);
try
{
var file = await DownloadWithRetryAsync(url, targetFolder, cancellationToken).ConfigureAwait(false);
downloaded.Add(file);
}
catch (OperationCanceledException) { throw; }
catch (Exception ex)
{
_log.LogWarning(ex, "Failed to download {Url}", Redact(url));
failed.Add(new DownloadFailure(url, ex.Message));
}
finally
{
gate.Release();
}
});
await Task.WhenAll(tasks).ConfigureAwait(false);
return new DownloadResult(downloaded.ToArray(), failed.ToArray());
}
// ---------------------------------------------------------------- Extraction
/// <summary>
/// Yield every Uploads/File URL in the result, handling all the
/// polymorphic shapes Keyfax payloads exhibit:
/// 1. Fault may be a single object OR an array.
/// 2. Uploads.File may be a single object OR an array.
/// 3. Each File element may be a plain string OR an object containing
/// a #cdata-section property (depending on the JSON serialiser used
/// by the Keyfax server when converting from XML).
/// </summary>
private static IEnumerable<string> ExtractUploadUrls(JsonDocument result)
{
if (!result.RootElement.TryGetProperty("KeyfaxData", out var keyfaxData)) yield break;
if (!keyfaxData.TryGetProperty("Fault", out var faultProperty)) yield break;
foreach (var fault in EnumerateOneOrMany(faultProperty))
{
if (!fault.TryGetProperty("Uploads", out var uploads)) continue;
if (uploads.ValueKind != JsonValueKind.Object) continue;
if (!uploads.TryGetProperty("File", out var fileProperty)) continue;
foreach (var file in EnumerateOneOrMany(fileProperty))
{
var url = ExtractUrl(file);
if (!string.IsNullOrWhiteSpace(url)) yield return url;
}
}
}
private static IEnumerable<JsonElement> EnumerateOneOrMany(JsonElement element)
{
if (element.ValueKind == JsonValueKind.Array)
{
foreach (var item in element.EnumerateArray()) yield return item;
}
else if (element.ValueKind != JsonValueKind.Null && element.ValueKind != JsonValueKind.Undefined)
{
yield return element;
}
}
private static string ExtractUrl(JsonElement file) => file.ValueKind switch
{
JsonValueKind.String => file.GetString(),
JsonValueKind.Object when file.TryGetProperty("#cdata-section", out var c)
&& c.ValueKind == JsonValueKind.String => c.GetString(),
_ => null,
};
// ---------------------------------------------------------------- Download
private async Task<DownloadedFile> DownloadWithRetryAsync(
string url, string targetFolder, CancellationToken ct)
{
var attempts = 0;
while (true)
{
attempts++;
try
{
return await DownloadAttemptAsync(url, targetFolder, ct).ConfigureAwait(false);
}
catch (HttpRequestException ex) when (IsTransient(ex) && attempts < _options.MaxRetries)
{
var delay = TimeSpan.FromMilliseconds(200 * Math.Pow(2, attempts));
_log.LogInformation(
"Transient failure on {Url} (attempt {Attempt}); retrying in {DelayMs}ms",
Redact(url), attempts, delay.TotalMilliseconds);
await Task.Delay(delay, ct).ConfigureAwait(false);
}
catch (IOException) when (attempts < _options.MaxRetries)
{
// Truncated stream / partial download. Retry once.
await Task.Delay(TimeSpan.FromMilliseconds(200 * attempts), ct).ConfigureAwait(false);
}
}
}
private async Task<DownloadedFile> DownloadAttemptAsync(
string url, string targetFolder, CancellationToken ct)
{
using var response = await _http
.GetAsync(url, HttpCompletionOption.ResponseHeadersRead, ct)
.ConfigureAwait(false);
response.EnsureSuccessStatusCode();
var safeName = BuildSafeFilename(url, response.Content.Headers);
var targetPath = ResolveCollisionFreePath(targetFolder, safeName);
long? expectedLength = response.Content.Headers.ContentLength;
long actualLength = 0;
await using (var fs = new FileStream(
targetPath,
FileMode.CreateNew,
FileAccess.Write,
FileShare.None,
bufferSize: 81920,
useAsync: true))
await using (var src = await response.Content.ReadAsStreamAsync(ct).ConfigureAwait(false))
{
var buffer = new byte[81920];
int read;
while ((read = await src.ReadAsync(buffer.AsMemory(), ct).ConfigureAwait(false)) > 0)
{
await fs.WriteAsync(buffer.AsMemory(0, read), ct).ConfigureAwait(false);
actualLength += read;
}
}
if (expectedLength.HasValue && expectedLength.Value != actualLength)
{
// Partial download. Drop the file and surface the failure.
try { File.Delete(targetPath); } catch { /* best effort */ }
throw new IOException(
$"Download truncated: expected {expectedLength.Value} bytes, received {actualLength}.");
}
var contentType = response.Content.Headers.ContentType?.MediaType;
_log.LogInformation(
"Downloaded {Url} ({Bytes} bytes, {ContentType}) -> {Path}",
Redact(url), actualLength, contentType ?? "unknown", targetPath);
return new DownloadedFile(
SourceUrl: url,
LocalPath: targetPath,
BytesWritten: actualLength,
ContentType: contentType);
}
// ---------------------------------------------------------------- Filename
/// <summary>
/// Build a filesystem-safe filename. The original filename embedded in
/// the URL is treated as untrusted user input - we generate our own
/// GUID-prefixed name and preserve only a sanitised extension. The
/// original filename is not used as the on-disk name; callers wanting
/// to display it should persist it as metadata.
/// </summary>
private static string BuildSafeFilename(string url, HttpContentHeaders headers)
{
var originalName = TryGetOriginalName(url, headers);
var ext = SanitiseExtension(Path.GetExtension(originalName));
var prefix = Guid.NewGuid().ToString("N");
return string.IsNullOrEmpty(ext) ? prefix : prefix + ext;
}
private static string TryGetOriginalName(string url, HttpContentHeaders headers)
{
// Prefer Content-Disposition if Keyfax sends one.
var d = headers.ContentDisposition;
if (!string.IsNullOrEmpty(d?.FileNameStar)) return d.FileNameStar.Trim('\"');
if (!string.IsNullOrEmpty(d?.FileName)) return d.FileName.Trim('\"');
// Fall back to the 'f' query parameter (Keyfax-uploaded URLs use
// ?f=GUID/originalname.ext) or the URL path basename.
try
{
var uri = new Uri(url);
var f = HttpUtility.ParseQueryString(uri.Query).Get("f");
return string.IsNullOrEmpty(f) ? Path.GetFileName(uri.AbsolutePath) : Path.GetFileName(f);
}
catch (UriFormatException) { return string.Empty; }
}
private static string SanitiseExtension(string ext)
{
if (string.IsNullOrEmpty(ext)) return string.Empty;
ext = ext.ToLowerInvariant();
if (ext.Length > 8) ext = ext.Substring(0, 8);
return new string(ext.Where(c => c == '.' || char.IsLetterOrDigit(c)).ToArray());
}
private static string ResolveCollisionFreePath(string folder, string fileName)
{
var path = Path.Combine(folder, fileName);
var i = 1;
while (File.Exists(path))
{
var stem = Path.GetFileNameWithoutExtension(fileName);
var ext = Path.GetExtension(fileName);
path = Path.Combine(folder, $"{stem}_{i}{ext}");
i++;
}
return path;
}
// ---------------------------------------------------------------- Helpers
/// <summary>
/// Decide whether an HttpRequestException represents a transient error
/// worth retrying. 5xx responses, request timeouts, and transport-level
/// errors (DNS, TCP) are transient; 4xx responses (especially 404, 410,
/// 401, 403) are not.
/// </summary>
private static bool IsTransient(HttpRequestException ex)
{
if (ex.StatusCode is HttpStatusCode code)
{
if (code == HttpStatusCode.RequestTimeout) return true;
return (int)code >= 500 && (int)code < 600;
}
return true; // transport-level failure
}
/// <summary>
/// Treat the upload URL as sensitive when logging - the GUID embedded in
/// the URL grants access for the 24-hour window. Log only host and a
/// path-tail hint for diagnostic correlation.
/// </summary>
private static string Redact(string url)
{
try
{
var u = new Uri(url);
var tail = Path.GetFileName(u.AbsolutePath);
return string.IsNullOrEmpty(tail)
? $"{u.Scheme}://{u.Host}/..."
: $"{u.Scheme}://{u.Host}/.../{tail}";
}
catch
{
return "[unparseable]";
}
}
}
// ---------------------------------------------------------------- DTOs
public sealed class KeyfaxFileDownloaderOptions
{
/// <summary>Maximum number of concurrent downloads. Default 4.</summary>
public int MaxConcurrency { get; init; } = 4;
/// <summary>Maximum retry attempts for transient errors. Default 3.</summary>
public int MaxRetries { get; init; } = 3;
}
public sealed record DownloadedFile(
string SourceUrl,
string LocalPath,
long BytesWritten,
string ContentType);
public sealed record DownloadFailure(
string SourceUrl,
string Reason);
public sealed record DownloadResult(
IReadOnlyList<DownloadedFile> Downloaded,
IReadOnlyList<DownloadFailure> Failed)
{
public bool AllSucceeded => Failed.Count == 0;
}
// ---------------------------------------------------------------- Composition root
/// <summary>
/// Example wiring. The HttpClient is configured with a timeout suitable for
/// most uploads; sites with video-bearing KeyNect captures may need to raise
/// it. In production prefer IHttpClientFactory over raw HttpClient.
/// </summary>
public static class KeyfaxFileDownloaderRegistration
{
public static IServiceCollection AddKeyfaxFileDownloader(
this IServiceCollection services,
Action<KeyfaxFileDownloaderOptions> configure = null)
{
services.AddHttpClient<KeyfaxFileDownloader>(client =>
{
client.Timeout = TimeSpan.FromMinutes(2);
client.DefaultRequestHeaders.Accept.Add(
new MediaTypeWithQualityHeaderValue("*/*"));
});
var options = new KeyfaxFileDownloaderOptions();
configure?.Invoke(options);
services.AddSingleton(options);
return services;
}
}
// ---------------------------------------------------------------- Calling site
/// <summary>
/// Illustrative result-processing pipeline showing the downloader being
/// invoked synchronously during result retrieval - within the 24-hour file
/// availability window.
/// </summary>
public sealed class KeyfaxResultProcessor
{
private readonly KeyfaxFileDownloader _files;
private readonly ILogger<KeyfaxResultProcessor> _log;
private readonly string _uploadStoreRoot;
public KeyfaxResultProcessor(
KeyfaxFileDownloader files,
ILogger<KeyfaxResultProcessor> log,
string uploadStoreRoot)
{
_files = files;
_log = log;
_uploadStoreRoot = uploadStoreRoot;
}
public async Task ProcessAsync(string sessionGuid, string resultJson, CancellationToken ct)
{
using var doc = JsonDocument.Parse(resultJson);
var folder = Path.Combine(_uploadStoreRoot, sessionGuid);
var download = await _files.DownloadAllAsync(doc, folder, ct).ConfigureAwait(false);
if (!download.AllSucceeded)
{
// Surface failures to operations - do not silently move on. Either
// throw to retry the entire result-processing job, or persist the
// failures for manual investigation. Be aware of the 24-hour
// window when choosing a retry strategy.
_log.LogError(
"{Failed} of {Total} files failed to download for session {Guid}",
download.Failed.Count,
download.Failed.Count + download.Downloaded.Count,
sessionGuid);
}
// Persist the downloaded files alongside the host's repair record.
// ...
}
}
