Live Streaming Tutorial

TROUBLESHOOTING

1. Publisher Side

There are 4 major areas which have plenty of failure potential:

1.1 Capture Devices

Most publishers have a fantastic setup of production equipment – the “Capture Devices.” This is the area with the least amount of failure potential among experienced production houses and broadcasters.

1.2 Encoder

There are typically 3 types of encoders: Lightcast Encoders, Third-Party Hardware Encoders, Self-Built Encoders.

• Lightcast Encoders: Publishers who use a Lightcast Encoder have 0 liability in this area and an extremely low risk of failure in this area.
• Third-Party Hardware Encoders: Publishers who invested into a professionally built and configured third-party hardware encoder have a very low risk of failure in this area, once the encoder has been configured correctly.
• Self-Built Encoders (often the cheapest entry-level solution): Publishers using a software encoder on a PC or Mac (a self-built encoder setup) are affected the strongest in this area and bound to troubleshoot their encoders on a regular basis. Even small, unnoticed software changes (in the operating system, drivers, software updates, software for other use of the computer, etc…), can have an immediate effect on the performance of the software encoder, or the input and output hardware connection. If you plan to use a self-built encoder, ensure that your team includes staff with the ncessary expertise to build, maintain, and troubleshoot the setup. This expertise should be readily available at all times to address any issues that may arise.

1.3 Local Network

This area often bears underestimated risks with plenty of failure potential. It starts with the connection: how is your encoder connected? Via cable or via WiFi? What is the setup of your LAN (computer) network? Your network may include: switches, hubs, routers, cables, plugs, wall-sockets. Have a professional system administrator setup, maintain and check on your local computer network.

1.4 Internet Service Provider

Often a very frustrating area of failure potential, as it is beyond anyone’s control and ISPs are often unreachable, or customer support unable to help, or unknowledgeable in regards to live-streaming requirements. In most cases, anyone you can reach at an ISP is completely unaware of possible connection issues within their network. As the signal of your live-stream leaves your building, it can travel to the CDN’s access point (the CDN’s POP location where you decided to ingest your live-stream into the CDN) through different routes. Along the route of the signal there are different hubs and proxy servers with potential bottle-necks and connection problems. Your ISP most likely does not own the cable networks along the entire way, but usually has to pass the signal along to third-party cable networks. There are usually several hubs along the way which can hold up the flow of the signal. Pulling a trace-route shows you all hubs along the way and the delay times of each hub. It will allow you to detect the bottle-necks where the signal is being slowed down too much in order to ensure good streaming performance. You should send the trace-route data to the highest possible level of tech-support you can get a hold of at your Internet Service Provider and put enough pressure on them to be willing to work on eliminating the delays of the signal flow.

2. CDN Side

You may wonder why “Server Uptime” is the only item on the list for this area. The reason is simply that your streaming service provider simply “mirrors” the live-stream signal it receives from your encoder. It usually does not change the format, resolution, quality, codec of the stream, but simply makes it available to a multitude of concurrent viewers. Therefore, the uptime of the streaming servers is your CDN's foremost responsibility and focus.

If the streaming servers go down, your stream will be down – entirely. There is no “grey zone.” Your live-stream is either up or it isn’t if your streaming provider’s servers are down. It is therefore crucial not to use a single-point streaming provider, but a multi-site CDN (Content Delivery Network), which is a cloud-server-network with multiple server locations. The redundancies within a CDN ensure the highest possible level of uptime guarantee. The more server locations a CDN provides (AKA “nodes”), the lower the risk for a network-wide “meltdown.”

There are generally two types of CDNs (“streaming-cloud-networks”): regional CDNs and global CDNs. Regional CDNs have multiple server locations within a region (country or continent), with up to 5 locations (nodes) usually. Global CDNs are larger with usually more than 10 locations (nodes) distributed around the globe for increased stability and performance. The probability of a server-side downtime is close to zero if you are using a global CDN.

3. User Side

There are 5 major factors which can cause usability issues on the user’s end:

3.1 Device

Given the vast array of devices, browsers, operating systems, and protocol standards, it is simply impossible to guarantee 100% compatibility with every device and software setup on the planet. However, it has to be the highest goal to achieve the maximum of device compatibility. This is why using the correct codec and following the live-streaming setup guide is crucial. That’s about all you can do. There will always be viewers whose devices encounter issues, whether due to outdated software, compatibility conflicts, hardware limitations, or an inability to connect to or maintain a live-stream. Despite all measures taken to keep this number as low as possible, a certain remnant of users with device issues is to be expected.

3.2 Local Network

Just like on your side as a publisher, the user’s connection is subject to a local network, too. Although less extensive, each user has a local hardware and software setup which may include cable connections, router, WiFi hub.

3.3 Cell Phone Networks

Cell Phone Networks are among the most volatile ways to connect to the internet, thus among the riskiest types of connections to watch a live-stream. While it may be possible in many cases, being subject to many different influences, a data connection of a cell phone network can easily drop from one second to the next. This is especially true on the road while between cell phone towers, in certain regions with limited coverage of a particular network, or in busy places where a network is used by a multitude of clients at the same time.

3.4 Internet Service Provider

They often tend to be large cable companies, spoiled with tens of thousands of clients, or with a monopoly in a specific region. In many cases, it is very hard to receive a decent level of knowledgeable customer care. Your viewers are subject to the same need for reliability and stability of the local ISPs cable networks, hubs, connections, proxy servers and signal routes, as you are as a publisher on your end. Small glitches in the network, a temporary slow-down, or a hardware update somewhere within the ISP’s cable system, can interrupt a viewer’s internet connection just enough to lose the connection to a live-stream. If this happens sequentially, it can – understandably – lead to frustrations on the viewer’s end.

3.5 Knowledge

Not every user who attempts to view a live-stream has the same knowledge-level in the area of technology, computers, smart phone devices, tablets, or local WiFi and internet connections. It is important to understand the possible limitations in the area of tech-skills and knowledge. In some cases, it may be enough to point to the user manuals of devices used. In some cases, the viewer may need assistance from a tech-savvy individual to troubleshoot the issue or might need to contact his/her local ISP for support.

4. Final Summary and Troubleshooting Advice

Unlike with areas 1 and 2, area 3 the “User Side” only results in singular cases. If anything happens on the publisher-side or on the CDN-side, the streams are down for everyone.

This is how you can distinguish whether a reported viewer-issue is on the viewer’s end, or on yours or your CDN’s side:

• Ideal Case: you cannot see (reproduce) the issue your viewer reports.
  Cause: the issue is on the user side.
  Solution: you might be able to assist him/her by pointing out the different possible root causes for the problem (as described above under “User Side”).
  Why is this case “ideal”?: because it only affects a limited number of viewers.

• Not-Ideal Case: you can see (reproduce) the issue your viewer reports, but the streams are generally up and running. The streams just look like they are “jumping” or “buffer” frequently.
  Cause: the issue is on your side (the “Publisher Side”).
  Solution: read through the possible causes as described above under “Publisher Side,” and systematically rule out each potential source of conflict step-by-step.
  Why is this case “not so ideal”?: because it affects all viewers, but at least you can do something about it. The cause is in an area you have control over – which gives you a crucial advantage.

• Worst Case: you can see (reproduce) the issue your viewer reports, but the streams are entirely down long-term. The players remain black.
  Cause: the issue is either on the CDN side or your side (in case your encoder establishes no connection at all, or does not broadcast a valid signal).
  Solution: check that your encoder is connected to a valid access point (POP) of the LightcastCDN and that it is sending a valid signal (correct codecs and your encoder is configured according to the Lightcast Livestreaming Manual). Open a Support Ticket in the Lightcast Support Center first, then give us a call right after. Why? The Support Ticket will go straight to the correct person at Lightcast – an engineer who can actually do something about it. Calling first may delay the process, as your message must be relayed to the appropriate department. It is crucial that Lightcast Tech Support receives your ticket directly from you, along with as much detailed technical information as possible, rather than relying on a receptionist to convey your issue. Support Tickets instantly show up on the screen of the appropriate person and can instantly be worked on.
  Why is this case a “worst case”?: because it affects all viewers and it is something outside of your, or your ISP's, areas of control. That said, this is the least likely case, as the CDN’s internal redundancies and automated signal routing, as well as the number of servers and nodes, is among the most solid on the planet today. It is a Tier-1, high-performance, high-security CDN and the downtime is practically 0.