Recherche avancée

Sur d’autres sites (4241)

• The Guide to an Ethical Web : With Big Data Comes Big Responsibility

  13 mars, par Alex Carmona

  Roughly two-thirds of Earth’s 8 billion people use the internet for communication, education, entertainment, business and more. We are connected globally in ways previous generations could’ve never dreamed of. It’s been a wild ride, and we’re just starting.

  Many users have learned that experiences online can be a mix of good and bad. Sometimes, the bad can feel like it outweighs the good, particularly when large tech companies use our data shadily, cut corners on accessibility or act in any other way that devalues the human being behind the screen.

  As fellow internet citizens, what responsibility do we have to create a more ethical web for our customers ?

  In this article, we’ll look at ethical principles online and how to act (and not act) to build trust, reach customers regardless of ability, safeguard privacy and stay compliant while improving business outcomes.

  

  What is an “ethical web” ?

  When we talk about the ethical web, we’re talking about the use of the internet in an ethical way. Among other values, it involves transparency, consent and restraint. It applies the Golden Rule to the internet : Treat others (and their data and user experience) how you’d want yourself (and yours) to be treated. 

  With limited oversight, the internet has evolved in ways that often prioritise profit over user rights. While selling data or pushing cookies might seem logical in this context, they can undermine trust and reputation. And the tide is slowly but surely shifting as consumers and legislators push back.

  Consumers no longer want to buy from companies that will use their data in ways they don’t agree to. In 2022, 75% of UK and US consumers surveyed said they were uncomfortable purchasing from businesses with weak data ethics.

  Legislators worldwide have been taking part in this effort for nearly a decade, with laws like GDPR in the EU and LGPD in Brazil, as well as the various state laws in the US, like California’s CCPA and Virginia’s VCDPA. 

  Even tech giants are no longer above the law, like Meta, which was fined over a billion Euros for GDPR violations in 2023.

  

  These changes may make the internet feel less business-friendly at first glance, but ethical choices ultimately build a stronger digital ecosystem for both companies and consumers. 

  Likewise, all internet users alike can make this happen by shunning short-term profit and convenience for healthier, long-term choices and behaviour.

  As we dig into what it takes to build an ethical web, remember that no company or individual is free from mistakes in these areas nor is it an overnight fix. Progress is made one click at a time.

  Ethical SEO : Optimising your content and your ethics

  Content creation and search engine optimisation (SEO) require so much work that it’s hard to fault creators for not always abiding by search engine guidelines and seeking shortcuts – especially when there’s a sea of LinkedIn posts about how copying/pasting ChatGPT responses helped someone rank #1 for several keywords in one week.

  However, users turn to Google and other search engines for something of substance that will guide or entertain them.

  Content meets customer needs and is more likely to lead to sales when it’s well-written, original and optimised just enough to make it easier to find on the first page of results. This doesn’t happen when content teams dilute quality and waste a reader or viewer’s time on posts that will only yield a higher bounce rate.

  Some SEO pros do find success by building backlinks through private blog networks or crafting a million unedited posts with generative AI, but it’s short-lived. Google and other search engines always catch up, and their content plummets or gets penalised and delisted with every new update.

  Content teams can still rank at the top while sticking to ethical SEO principles. Here’s a sample list of dos and don’ts to get started :

  • Do put content quality above all else. Make content that serves the audience, not just a brand or partner ad network.
  • Do apply the E-E-A-T framework. Search engines value content written by authors who bring expertise, experience, authority and trust (E-E-A-T).
  • Don’t keyword stuff. This might have worked in the early days of SEO, but it hurts readability and now harms article performance.
  • Do use alt text as intended. While it can still help SEO, alt text should prioritise accessibility for users with screen readers.

  • Don’t steal content. Whether it’s violating copyright, copying/pasting other people’s content or simply paraphrasing without citation, companies should never steal content.
  • Don’t steal ideas. It’s okay to join in on a current conversation or trends in an industry, but content creators should be sure they have something valuable to add.
  • Do use AI tools as partners, not creators. AI can be an incredible aid in crafting content, but it should never be posted without a human’s touch.

  When we follow ethical SEO guidelines and get more clients with our content, how do we best handle their data ?

  Ethical data governance : Important principles and how to avoid data misuse

  Data governance comprises every aspect of how a company manages data, including storage, security, privacy, lifecycle management, setting policies and maintaining compliance with laws like GDPR and HIPAA.

  Applying data ethics to governance is doing it all in a transparent, restrained way that acknowledges an individual’s right to ownership over their data. 

  For organisations, this translates to getting consent to collect data and clearly spelling out how it will be stored and used — and sticking to it.

  If a user’s birth date is needed for legal reasons, it cannot be sold to a third party or later used for something else without explicit permission. Reusing data in ways that stray from its original purpose is a form of commingling, one of the data misuses that is easy for even well-intentioned teams to do accidentally.

  Ethical data governance also includes the vigilant safeguarding of users’ data and minimising potential privacy issues.

  Failing to implement and adhere to strong security measures leads to situations like the National Public Data (NPD) breach, where cyber criminals expose the addresses, phone numbers and social security numbers of hundreds of millions of people. This was due in large part to a weakness in storing login credentials and a lack of password policy enforcement.

  No one at NPD wanted this to happen, but security likely took a backseat to other business concerns, leading to the company’s filing for bankruptcy.

  More importantly, as a data broker that aggregates information from other sources, the people affected likely had no clue this organisation had been buying and selling their data. The companies originally entrusted with their information helped provide the leaked data, showing a lack of care for privacy.

  Situations like this reinforce the need for strict data protection laws and for companies to refine their data governance approach. 

  Businesses can improve their data governance posturing with managers and other higher-ups setting the right tone at the top. If leadership takes a firm and disciplined approach by setting and adhering to strong policies, the rest of the team will follow and minimise the chances of data misuse and security incidents.

  One way to start is by using tools that make the principles of data ethics easier to follow.

  Ethical web analytics : Drawing insights while respecting privacy

  Web analytics tools are designed to gather data about users and what they do while visiting a site.
  The most popular tool worldwide is Google Analytics (GA). Its brand name and feature set carry a lot of weight, but many former users have switched to alternatives due to dissatisfaction with the changes made in GA4 and reservations about the way Google handles data.

  

  Google is another tech giant that has been slapped with massive GDPR fines for issues over its data processing practices. It has run so afoul of compliance that it was banned in France and Austria for a while. Additionally, in the US Department of Justice’s ongoing antitrust lawsuit against Google, the company’s data tracking has been targeted for both how it affects users and potential rivals.

  Unlike GA, ethical web analytics tools allow websites to get the data they need while respecting user privacy.

  Matomo offers privacy protections like :

  • Providing data anonymisation and IP anonymisation
  • Allowing users the ability to not process personally identifiable information (PII)
  • Disabling the ability to track users across websites by default to make compliance easier
  • Giving users full control over their customer data without the risk of it getting into the hands of third parties
  • Much more

  We’re also fully transparent about how we handle your data on the web and in the Matomo Cloud and in how we build Matomo as an open-source tool. Our openness allows you to be more open with your customers and how you ethically use their data.

  There are other GDPR-compliant tools on the market, but some of them, like Adobe Analytics, require more setup from users for compliance, don’t grant full control over data and don’t offer on-premise options or consent-free tracking.

  Beyond tracking, there are other ways to make a user’s experience more enjoyable and ethical.

  Ethical user experience : User-friendliness, not user-hostility

  When designing a website or application, creating a positive user experience (UX) always comes first. 

  The UI should be simple to navigate, data and privacy policy information should be easy to find and customers should feel welcomed. They must never be tricked into consenting or installing. 

  When businesses resort to user-hostile tactics, the UX becomes a battle between the user and them. What may seem like a clever tactic to increase sign-ups can alienate potential customers and ruin a brand’s image. 

  Here are some best practices for creating a more ethical UX :

  Avoid dark patterns

  Dark patterns are UI designs and strategies that mislead users into paying for, agreeing to or doing something they don’t actually want. These designs are unethical because they’re manipulative and remove transparency and consent from the interaction. 

  In some cases, they’re illegal and can bring lawsuits. 

  In 2023, Italy’s Data Protection Authority (DPA) fined a digital marketing company €300,000 for alleged GDPR violations. They employed dark patterns by asking customers to accept cookies again after rejecting them and placing the option to reject cookies outside the cookie banner. 

  Despite their legality and 56% of surveyed customers losing trust in platforms that employ dark patterns, a review by the Organisation for Economic Co-operation and Development (OECD) found that 76% of the websites examined contained at least one dark pattern.

  

  If a company is worried that they may be relying on dark patterns, here are some examples of what to avoid :

  • Pre-ticking boxes to have users agree to third-party cookies, sign up for a newsletter, etc.
  • Complicated cookie banners without a one-click way to reject all unnecessary cookies
  • Hiding important text with text colour, under drop-down menus or requiring hovering over something with a mouse 
  • “Confirm shaming” users with emotionally manipulative language to delay subscription cancellations or opt out of tracking 

  Improve trust centres

  Trust centres are the sections of a website that outline how a company approaches topics like data governance, user privacy and security. 

  They should be easy to find and understand. If a user has a question about a company’s data policy, it should be one click away with language that doesn’t require a law degree to comprehend.

  Additionally, trust centres must cover all relevant details, including where data is stored and who does the subprocessing. This is an area where even some of the best-intentioned companies may miss the mark, but it’s also an easy fix and a great place to start creating a more ethical web.

  Embrace inclusivity

  People want to feel welcomed to the party — and deserve to be — regardless of their race, ethnicity, religion, gender identity, orientation or ability. 

  Inclusivity is great for customers and companies alike. 

  A study by the Unstereotype Alliance found that progressive marketing drove up short- and long-term sales, customer loyalty and purchase consideration. A Kantar study reported that 75% of surveyed customers around the world consider a company’s diversity and inclusivity when making a purchasing decision.

  An easy place to start embracing inclusivity is with a website’s blog images. The people in photos and cartoons should reflect a variety of different backgrounds.

  Another area to improve inclusivity is by making your site or app more accessible.

  Accessibility ethics : An internet for everyone

  Accessibility is designing your product in a way that everyone can enjoy or take part in, regardless of ability. Digital accessibility is applying this design to the web and applications by making accommodations like adding descriptive alt text to images for users with visual impairments.

  Just because someone has a hearing, vision, speech, mobility, neurological or other impairment doesn’t mean they have any less of a right to shop online, read silly listicles or get into arguments with strangers in the comment section.

  Beyond being the right thing to do, the Fable team shows there’s a strong business case for accessibility. People with disabilities have money to spend, and the accommodations businesses make for them often benefit people without disabilities, too – as anyone who streams with subtitles can attest.

  Despite being a win-win for greater inclusivity and business, much of the web is still inaccessible. WebAIM, a leader in web accessibility, studied a million web pages and found an average of over 55 accessibility errors per page.

  We must all play a more active role in improving the experience of our users with disabilities, and we can start with accessibility auditing and testing.

  An accessibility audit is an evaluation of how usable a site is for people with disabilities. It may be done in-house by an expert on a company’s team or, for better results, a third-party consultant who can give a fully objective audit.

  Auditing might consist of running an automated tool or manually checking your site, PDFs, emails and other materials for compliance with the Web Content Accessibility Guidelines list.

  Accessibility testing is narrower than auditing. It checks how accessibility or its absence looks in action. It can be done after a site, app, email or product is released, but it ideally starts in the development process.

  Testing should be done manually and with automated tools. Manual checks put developers in the position of their users, allowing them to get a better idea of what users are dealing with firsthand. Automated tools can save time and money, but there should always be manual testing in the process.

  Auditing gives teams an idea of where to start with improving accessibility, and testing helps make sure accommodations work as intended.

  Conclusion

  At Matomo, we strive to make the ethical web a reality, starting with web analytics.

  For our users, it means full compliance with stringent policies like GDPR and providing 100% accurate data. For their customers, it’s collecting only the data required to do the job and enabling cookieless configurations to get rid of annoying banners. 

  For both parties, it’s knowing that respect for privacy is one of our foundational values, whether it’s the ability to look under Matomo’s hood and read our open-source code, the option to store data on-premise to minimise the chances of it falling into the wrong hands or one of the other ways that we protect privacy.

  If you weren’t 100% ethical before, it’s never too late to change. You can even bring your Google Analytics data with you.

  Join us in our mission to improve the web. We can’t do it alone ! 

  Try Matomo free for 21 days

  no credit card required

• How to resize dimensions of video through ffmpeg-python ?

  25 janvier, par kunambi

  I'm trying to resize a video file which a user has uploaded to Django, by using ffmpeg-python. The documentation isn't very easy to understand, so I've tried to cobble this together from various sources.

  


  This method is run in a celery container, in order to not slow the experience for the user. The problem I'm facing is that I can't seem to resize the video file. I've tried two different approaches :

  


  from django.db import models
from io import BytesIO
from myapp.models import MediaModel


def resize_video(mypk: str) -> None:
    instance = MediaModel.objects.get(pk=mypk)
    media_instance: models.FileField = instance.media
    media_output = "test.mp4"
    buffer = BytesIO()

    for chunk in media_instance.chunks():
        buffer.write(chunk)

    stream_video = ffmpeg.input("pipe:").video.filter("scale", 720, -1)  # resize to 720px width
    stream_audio = ffmpeg.input("pipe:").audio
    process = (
        ffmpeg.output(stream_video, stream_audio, media_output, acodec="aac")
        .overwrite_output()
        .run_async(pipe_stdin=True, quiet=True)
    )
    buffer.seek(0)
    process_out, process_err = process.communicate(input=buffer.getbuffer())
    # (pdb) process_out
    # b''

    # attempting to use `.concat` instead
    process2 = (
        ffmpeg.concat(stream_video, stream_audio, v=1, a=1)
        .output(media_output)
        .overwrite_output()
        .run_async(pipe_stdin=True, quiet=True)
    )
    buffer.seek(0)
    process2_out, process2_err = process2.communicate(input=buffer.getbuffer())
    # (pdb) process2_out
    # b''


  


  As we can see, no matter which approach chosen, the output is an empty binary. The process_err and process2_err both generate the following message :

  


  ffmpeg version N-111491-g31979127f8-20230717 Copyright (c) 2000-2023 the
FFmpeg developers
  built with gcc 13.1.0 (crosstool-NG 1.25.0.196_227d99d)
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static
--pkg-config=pkg-config --cross-prefix=x86_64-w64-mingw32- --arch=x86_64
--target-os=mingw32 --enable-gpl --enable-version3 --disable-debug
--disable-w32threads --enable-pthreads --enable-iconv --enable-libxml2
--enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp
--enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl
--disable-libpulse --enable-libvmaf --disable-libxcb --disable-xlib
--enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth
--enable-chromaprint --enable-libdav1d --enable-libdavs2
--disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r
--enable-libgme --enable-libkvazaar --enable-libass --enable-libbluray
--enable-libjxl --enable-libmp3lame --enable-libopus --enable-librist
--enable-libssh --enable-libtheora --enable-libvpx --enable-libwebp
--enable-lv2 --enable-libvpl --enable-openal --enable-libopencore-amrnb
--enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg
--enable-libopenmpt --enable-librav1e --enable-librubberband
--enable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt
--enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --disable-libdrm
--disable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc
--enable-libplacebo --enable-libx264 --enable-libx265 --enable-libxavs2
--enable-libxvid --enable-libzimg --enable-libzvbi
--extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags=
--extra-ldflags=-pthread --extra-ldexeflags= --extra-libs=-lgomp
--extra-version=20230717
  libavutil      58. 14.100 / 58. 14.100
  libavcodec     60. 22.100 / 60. 22.100
  libavformat    60. 10.100 / 60. 10.100
  libavdevice    60.  2.101 / 60.  2.101
  libavfilter     9.  8.102 /  9.  8.102
  libswscale      7.  3.100 /  7.  3.100
  libswresample   4. 11.100 /  4. 11.100
  libpostproc    57.  2.100 / 57.  2.100
 "Input #0, mov,mp4,m4a,3gp,3g2,mj2, frompipe:':\r\n"
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: mp42mp41
    creation_time   : 2020-11-10T15:01:09.000000Z
  Duration: 00:00:04.16, start: 0.000000, bitrate: N/A
  Stream #0:0[0x1](eng): Video: h264 (Main) (avc1 / 0x31637661),
yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 2649 kb/s, 25 fps, 25
tbr, 25k tbn (default)
    Metadata:
      creation_time   : 2020-11-10T15:01:09.000000Z
      handler_name    : ?Mainconcept Video Media Handler
      vendor_id       : [0][0][0][0]
      encoder         : AVC Coding
  Stream #0:1[0x2](eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz,
stereo, fltp, 317 kb/s (default)
    Metadata:
      creation_time   : 2020-11-10T15:01:09.000000Z
      handler_name    : #Mainconcept MP4 Sound Media Handler
      vendor_id       : [0][0][0][0]
Stream mapping:
  Stream #0:0 (h264) -> scale:default (graph 0)
  scale:default (graph 0) -> Stream #0:0 (libx264)
  Stream #0:1 -> #0:1 (aac (native) -> aac (native))
[libx264 @ 00000243a23a1100] using SAR=1/1
[libx264 @ 00000243a23a1100] using cpu capabilities: MMX2 SSE2Fast SSSE3
SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 00000243a23a1100] profile High, level 3.0, 4:2:0, 8-bit
[libx264 @ 00000243a23a1100] 264 - core 164 - H.264/MPEG-4 AVC codec -
Copyleft 2003-2023 - http://www.videolan.org/x264.html - options: cabac=1
ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00
mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11
fast_pskip=1 chroma_qp_offset=-2 threads=6 lookahead_threads=1
sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0
constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1
weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40
intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0
qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
 "Output #0, mp4, toaa37f8d7685f4df9af85b1cdcd95997e.mp4':\r\n"
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: mp42mp41
    encoder         : Lavf60.10.100
  Stream #0:0: Video: h264 (avc1 / 0x31637661), yuv420p(tv, progressive),
800x450 [SAR 1:1 DAR 16:9], q=2-31, 25 fps, 12800 tbn
    Metadata:
      encoder         : Lavc60.22.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
  Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo,
fltp, 128 kb/s (default)
    Metadata:
      creation_time   : 2020-11-10T15:01:09.000000Z
      handler_name    : #Mainconcept MP4 Sound Media Handler
      vendor_id       : [0][0][0][0]
      encoder         : Lavc60.22.100 aac
frame=    0 fps=0.0 q=0.0 size=       0kB time=N/A bitrate=N/A
speed=N/A    \r'
frame=   21 fps=0.0 q=28.0 size=       0kB time=00:00:02.75 bitrate=  
0.1kbits/s speed=4.75x    \r'
[out#0/mp4 @ 00000243a230bd80] video:91kB audio:67kB subtitle:0kB other
streams:0kB global headers:0kB muxing overhead: 2.838559%
frame=  104 fps=101 q=-1.0 Lsize=     162kB time=00:00:04.13 bitrate=
320.6kbits/s speed=4.02x    
[libx264 @ 00000243a23a1100] frame I:1     Avg QP:18.56  size:  2456
[libx264 @ 00000243a23a1100] frame P:33    Avg QP:16.86  size:  1552
[libx264 @ 00000243a23a1100] frame B:70    Avg QP:17.55  size:   553
[libx264 @ 00000243a23a1100] consecutive B-frames:  4.8% 11.5% 14.4%
69.2%
[libx264 @ 00000243a23a1100] mb I  I16..4: 17.3% 82.1%  0.6%
[libx264 @ 00000243a23a1100] mb P  I16..4:  5.9% 15.2%  0.4%  P16..4: 18.3% 
0.9%  0.4%  0.0%  0.0%    skip:58.7%
[libx264 @ 00000243a23a1100] mb B  I16..4:  0.8%  0.3%  0.0%  B16..8: 15.4% 
1.0%  0.0%  direct: 3.6%  skip:78.9%  L0:34.2% L1:64.0% BI: 1.7%
[libx264 @ 00000243a23a1100] 8x8 transform intra:68.2% inter:82.3%
[libx264 @ 00000243a23a1100] coded y,uvDC,uvAC intra: 4.2% 18.4% 1.2% inter:
1.0% 6.9% 0.0%
[libx264 @ 00000243a23a1100] i16 v,h,dc,p: 53% 25%  8% 14%
[libx264 @ 00000243a23a1100] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 19%  6% 70%  1% 
1%  1%  1%  0%  0%
[libx264 @ 00000243a23a1100] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 46% 21% 15%  2% 
5%  4%  3%  3%  1%
[libx264 @ 00000243a23a1100] i8c dc,h,v,p: 71% 15% 13%  1%
[libx264 @ 00000243a23a1100] Weighted P-Frames: Y:30.3% UV:15.2%
[libx264 @ 00000243a23a1100] ref P L0: 46.7%  7.5% 34.6%  7.3%  3.9%
[libx264 @ 00000243a23a1100] ref B L0: 88.0% 10.5%  1.5%
[libx264 @ 00000243a23a1100] ref B L1: 98.1%  1.9%
[libx264 @ 00000243a23a1100] kb/s:177.73
[aac @ 00000243a23a2e00] Qavg: 1353.589


  


  I'm at a loss right now, would love any feedback/solution.

  


• Adventures In NAS

  1er janvier, par Multimedia Mike — General

  In my post last year about my out-of-control single-board computer (SBC) collection which included my meager network attached storage (NAS) solution, I noted that :

  I find that a lot of my fellow nerds massively overengineer their homelab NAS setups. I’ll explore this in a future post. For my part, people tend to find my homelab NAS solution slightly underengineered.

  So here I am, exploring this is a future post. I’ve been in the home NAS game a long time, but have never had very elaborate solutions for such. For my part, I tend to take an obsessively reductionist view of what constitutes a NAS : Any small computer with a pool of storage and a network connection, running the Linux operating system and the Samba file sharing service.

  
  
  

  Many home users prefer to buy turnkey boxes, usually that allow you to install hard drives yourself, and then configure the box and its services with a friendly UI. My fellow weird computer nerds often buy cast-off enterprise hardware and set up more resilient, over-engineered solutions, as long as they have strategies to mitigate the noise and dissipate the heat, and don’t mind the electricity bills.

  If it works, awesome ! As an old hand at this, I am rather stuck in my ways, however, preferring to do my own stunts, both with the hardware and software solutions.

  My History With Home NAS Setups
  In 1998, I bought myself a new computer — beige box tower PC, as was the style as the time. This was when normal people only had one computer at most. It ran Windows, but I was curious about this new thing called “Linux” and learned to dual boot that. Later that year, it dawned on me that nothing prevented me from buying a second ugly beige box PC and running Linux exclusively on it. Further, it could be a headless Linux box, connected by ethernet, and I could consolidate files into a single place using this file sharing software named Samba.
  

  I remember it being fairly onerous to get Samba working in those days. And the internet was not quite so helpful in those days. I recall that the thing that blocked me for awhile was needing to know that I had to specify an entry for the Samba server machine in the LMHOSTS (Lanman hosts) file on the Windows 95 machine.

  However, after I cracked that code, I have pretty much always had some kind of ad-hoc home NAS setup, often combined with a headless Linux development box.

  In the early 2000s, I built a new beige box PC for a file server, with a new hard disk, and a coworker tutored me on setting up a (P)ATA UDMA 133 (or was it 150 ? anyway, it was (P)ATA’s last hurrah before SATA conquered all) expansion card and I remember profiling that the attached hard drive worked at a full 21 MBytes/s reading. It was pretty slick. Except I hadn’t really thought things through. You see, I had a hand-me-down ethernet hub cast-off from my job at the time which I wanted to use. It was a 100 Mbps repeater hub, not a switch, so the catch was that all connected machines had to be capable of 100 Mbps. So, after getting all of my machines (3 at the time) upgraded to support 10/100 ethernet (the old off-brand PowerPC running Linux was the biggest challenge), I profiled transfers and realized that the best this repeater hub could achieve was about 3.6 MBytes/s. For a long time after that, I just assumed that was the upper limit of what a 100 Mbps network could achieve. Obviously, I now know that the upper limit ought to be around 11.2 MBytes/s and if I had gamed out that fact in advance, I would have realized it didn’t make sense to care about super-fast (for the time) disk performance.

  At this time, I was doing a lot for development for MPlayer/xine/FFmpeg. I stored all of my multimedia material on this NAS. I remember being confused when I was working with Y4M data, which is raw frames, which is lots of data. xine, which employed a pre-buffering strategy, would play fine for a few seconds and then stutter. Eventually, I reasoned out that the files I was working with had a data rate about twice what my awful repeater hub supported, which is probably the first time I came to really understand and respect streaming speeds and their implications for multimedia playback.

  Smaller Solutions
  For a period, I didn’t have a NAS. Then I got an Apple AirPort Extreme, which I noticed had a USB port. So I bought a dual drive brick to plug into it and used that for a time. Later (2009), I had this thing called the MSI Wind Nettop which is the only PC I’ve ever seen that can use a CompactFlash (CF) card for a boot drive. So I did just that, and installed a large drive so it could function as a NAS, as well as a headless dev box. I’m still amazed at what a low-power I/O beast this thing is, at least when compared to all the ARM SoCs I have tried in the intervening 1.5 decades. I’ve had spinning hard drives in this thing that could read at 160 MBytes/s (‘dd’ method) and have no trouble saturating the gigabit link at 112 MBytes/s, all with its early Intel Atom CPU.

  Around 2015, I wanted a more capable headless dev box and discovered Intel’s line of NUCs. I got one of the fat models that can hold a conventional 2.5″ spinning drive in addition to the M.2 SATA SSD and I was off and running. That served me fine for a few years, until I got into the ARM SBC scene. One major limitation here is that 2.5″ drives aren’t available in nearly the capacities that make a NAS solution attractive.

  Current Solution
  My current NAS solution, chronicled in my last SBC post– the ODroid-HC2, which is a highly compact ARM SoC with an integrated USB3-SATA bridge so that a SATA drive can be connected directly to it :

  
  

  

  ODROID-HC2 NAS

  
  

  I tend to be weirdly proficient at recalling dates, so I’m surprised that I can’t recall when I ordered this and put it into service. But I’m pretty sure it was circa 2018. It’s only equipped with an 8 TB drive now, but I seem to recall that it started out with only a 4 TB drive. I think I upgraded to the 8 TB drive early in the pandemic in 2020, when ISPs were implementing temporary data cap amnesty and I was doing what a r/DataHoarder does.

  The HC2 has served me well, even though it has a number of shortcomings for a hardware set chartered for NAS :

  1. While it has a gigabit ethernet port, it’s documented that it never really exceeds about 70 MBytes/s, due to the SoC’s limitations
  2. The specific ARM chip (Samsung Exynos 5422 ; more than a decade old as of this writing) lacks cryptography instructions, slowing down encryption if that’s your thing (e.g., LUKS)
  3. While the SoC supports USB3, that block is tied up for the SATA interface ; the remaining USB port is only capable of USB2 speeds
  4. 32-bit ARM, which prevented me from running certain bits of software I wanted to try (like Minio)
  5. Only 1 drive, so no possibility for RAID (again, if that’s your thing)

  I also love to brag on the HC2’s power usage : I once profiled the unit for a month using a Kill-A-Watt and under normal usage (with the drive spinning only when in active use). The unit consumed 4.5 kWh… in an entire month.

  New Solution
  Enter the ODroid-HC4 (I purchased mine from Ameridroid but Hardkernel works with numerous distributors) :

  
  

  

  ODroid-HC4 with an SSD and a conventional drive

  
  

  I ordered this earlier in the year and after many months of procrastinating and obsessing over the best approach to take with its general usage, I finally have it in service as my new NAS. Comparing point by point with the HC2 :

  1. The gigabit ethernet runs at full speed (though a few things on my network run at 2.5 GbE now, so I guess I’ll always be behind)
  2. The ARM chip (Amlogic S905X3) has AES cryptography acceleration and handles all the LUKS stuff without breaking a sweat ; “cryptsetup benchmark” reports between 500-600 MBytes/s on all the AES variants
  3. The USB port is still only USB2, so no improvement there
  4. 64-bit ARM, which means I can run Minio to simulate block storage in a local dev environment for some larger projects I would like to undertake
  5. Supports 2 drives, if RAID is your thing

  How I Set It Up
  How to set up the drive configuration ? As should be apparent from the photo above, I elected for an SSD (500 GB) for speed, paired with a conventional spinning HDD (18 TB) for sheer capacity. I’m not particularly trusting of RAID. I’ve watched it fail too many times, on systems that I don’t even manage, not to mention that aforementioned RAID brick that I had attached to the Apple AirPort Extreme.

  I had long been planning to use bcache, the block caching interface for Linux, which can use the SSD as a speedy cache in front of the more capacious disk. There is also LVM cache, which is supposed to achieve something similar. And then I had to evaluate the trade-offs in whether I wanted write-back, write-through, or write-around configurations.

  This was all predicated on the assumption that the spinning drive would not be able to saturate the gigabit connection. When I got around to setting up the hardware and trying some basic tests, I found that the conventional HDD had no trouble keeping up with the gigabit data rate, both reading and writing, somewhat obviating the need for SSD acceleration using any elaborate caching mechanisms.

  Maybe that’s because I sprung for the WD Red Pro series this time, rather than the Red Plus ? I’m guessing that conventional drives do deteriorate over the years. I’ll find out.

  For the operating system, I stuck with my newest favorite Linux distro : DietPi. While HardKernel (parent of ODroid) makes images for the HC units, I had also used DietPi for the HC2 for the past few years, as it tends to stay more up to date.

  Then I rsync’d my data from HC2 -> HC4. It was only about 6.5 TB of total data but it took days as this WD Red Plus drive is only capable of reading at around 10 MBytes/s these days. Painful.

  For file sharing, I’m pretty sure most normal folks have nice web UIs in their NAS boxes which allow them to easily configure and monitor the shares. I know there are such applications I could set up. But I’ve been doing this so long, I just do a bare bones setup through the terminal. I installed regular Samba and then brought over my smb.conf file from the HC2. 1 by 1, I tested that each of the old shares were activated on the new NAS and deactivated on the old NAS. I also set up a new share for the SSD. I guess that will just serve as a fast I/O scratch space on the NAS.

  The conventional drive spins up and down. That’s annoying when I’m actively working on something but manage not to hit the drive for like 5 minutes and then an application blocks while the drive wakes up. I suppose I could set it up so that it is always running. However, I micro-manage this with a custom bash script I wrote a long time ago which logs into the NAS and runs the “date” command every 2 minutes, appending the output to a file. As a bonus, it also prints data rate up/down stats every 5 seconds. The spinning file (“nas-main/zz-keep-spinning/keep-spinning.txt”) has never been cleared and has nearly a quarter million lines. I suppose that implies that it has kept the drive spinning for 1/2 million minutes which works out to around 347 total days. I should compare that against the drive’s SMART stats, if I can remember how. The earliest timestamp in the file is from March 2018, so I know the HC2 NAS has been in service at least that long.

  For tasks, vintage cron still does everything I could need. In this case, that means reaching out to websites (like this one) and automatically backing up static files.

  I also have to have a special script for starting up. Fortunately, I was able to bring this over from the HC2 and tweak it. The data disks (though not boot disk) are encrypted. Those need to be unlocked and only then is it safe for the Samba and Minio services to start up. So one script does all that heavy lifting in the rare case of a reboot (this is the type of system that’s well worth having on a reliable UPS).

  Further Work
  I need to figure out how to use the OLED display on the NAS, and how to make it show something more useful than the current time and date, which is what it does in its default configuration with HardKernel’s own Linux distro. With DietPi, it does nothing by default. I’m thinking it should be able to show the percent usage of each of the 2 drives, at a minimum.

  I also need to establish a more responsible backup regimen. I’m way too lazy about this. Fortunately, I reason that I can keep the original HC2 in service, repurposed to accept backups from the main NAS. Again, I’m sort of micro-managing this since a huge amount of data isn’t worth backing up (remember the whole DataHoarder bit), but the most important stuff will be shipped off.

  The post Adventures In NAS first appeared on Breaking Eggs And Making Omelettes.