最近发现一些快手的作者，作品还不错，出于学习研究的目的，决定看一下怎么爬取数据。现在网上有一些爬虫工具，不过大部分都失效了，或者不开源。于是自己就写了一个小工具。先看一下成果：
软件只需要填写作者uid以及网页版的请求Cookie，即可实现自动下载，下载目录在程序根目录下的Download文件夹。
由于快手的风控比较厉害，软件也做了应对措施。不过需要用户点击软件中的提示文字，复制粘贴到浏览器，把请求的json保存到本地文件。使用软件提供的解析本地json按钮解析下载即可。如果返回的json文件很短或者没有数据，需要在快手的任意一个页面刷新一下，也就是告诉快手风控，现在是正常浏览，没有机器人的行为。
下面说一下构建整个App的思路。
1. 快手网页端准备
-
打开https://live.kuaishou.com/ ，在顶部搜索你要爬取的作者昵称，进入作者主页。也可以从App端分享作者的主页链接，粘贴进来。作者主页加载完成后，地址栏的地址一定要是类似：https://live.kuaishou.com/profile/xxxxxx。 后面的xxxxxx就是作者的user id。这个记住，复制出来，后面会用到。
-
按F12打开浏览器的开发者工具（我之前就说过开发者工具是好东西，研究爬虫必备，一定要好好学习）。
-
选择开发者工具顶部的“网络”，“全部”，如图所示。在请求列表中找到user id，点击它，右面就会出来请求的标头。里面有个Cookie，需要记住，复制出来。如果没有的话，记得刷新页面。
-
在列表里面可以看到很多请求，我们需要从中找到网页端展示作品列表的那条请求，即public开头的，或者直接在左上角搜索public，即可过滤绝大部分无关请求。这个请求的响应数据里面有作者作品的完整json响应。
你可以右击它，在新标签页面打开，打开后地址栏会显示完成的浏览器请求地址。这个网址需要记住，后续会用到。那个count默认是12或者20，我们用到时候，直接拉满，9999即可。
2. Postman拦截请求，模拟请求，并生成C#请求代码
-
安装postman interceptor拦截器，安装地址https://chromewebstore.google.com/detail/postman-interceptor/aicmkgpgakddgnaphhhpliifpcfhicfo 不得不说，这又是一个神器，搭配开发者工具，理论上可以搞定几乎所有的爬虫需求了。
-
打开Postman，点击右下角的Start Proxy，
开启拦截后，重新回到网页版作者主页，刷新一下页面，等页面加载完成后，点击停止拦截。否则列表会一直增多，因为他会拦截电脑的所有网络请求。这时Postman拦截器就会拦截到一大堆请求，同理，找到public请求，或者在左上角输入public，即可过滤出来我们需要的。
点击这个请求链接
这是Postman会打开一个新的窗口，包含了请求这个链接的所有参数以及标头信息。
点击Postman最右面的代码工具即可生成我们需要的代码。你可以选择C#、python、js、curl等等。
3. 使用WPF写界面以及下载逻辑
- 新建WPF工程，为了界面好看，这次我用了开源的WPF UI，之前用过HandyControl、MicaWPF，这些都是不错的UI控件库。
下载使用了开源的Downloader，请求使用了RestSharp，解析Json使用NewtonsoftJson，另外推荐一个免费的图标库FlatIcon。
界面如下：
点击查看代码
<ui:FluentWindow x:Class="KuaishouDownloader.MainWindow" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:d="http://schemas.microsoft.com/expression/blend/2008" xmlns:local="clr-namespace:KuaishouDownloader" xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:ui="http://schemas.lepo.co/wpfui/2022/xaml" Title="MainWindow" Width="900" Height="760" ExtendsContentIntoTitleBar="True" WindowBackdropType="Mica" WindowCornerPreference="Default" WindowStartupLocation="CenterScreen" mc:Ignorable="d"> <Grid> <Grid.RowDefinitions> <RowDefinition Height="Auto" /> <RowDefinition Height="*" /> </Grid.RowDefinitions> <ui:TitleBar Title="快手作者主页作品爬取" Height="32" /> <ui:Button x:Name="themeButton" Grid.Row="1" Width="32" Height="32" Margin="0,0,8,0" Padding="0" HorizontalAlignment="Right" VerticalAlignment="Top" Click="Theme_Click" CornerRadius="16" FontSize="24" Icon="{ui:SymbolIcon WeatherMoon48}" ToolTip="切换主题" /> <ui:SnackbarPresenter x:Name="snackbarPresenter" Grid.Row="1" VerticalAlignment="Bottom" /> <StackPanel Grid.Row="1" HorizontalAlignment="Center" VerticalAlignment="Center"> <Border Width="200" Height="200" HorizontalAlignment="Center" CornerRadius="100"> <ui:Image x:Name="imgHeader" Width="200" Height="200" CornerRadius="100" /> </Border> <ui:TextBlock x:Name="tbNickName" Margin="0,12,0,0" HorizontalAlignment="Center" /> <StackPanel Margin="0,12,0,0" Orientation="Horizontal"> <ui:TextBlock Width="60" Margin="0,12,0,0" VerticalAlignment="Center" Text="uid" /> <ui:TextBox x:Name="tbUid" Width="660" Height="36" VerticalContentAlignment="Center" ToolTip="App进入作者主页，分享主页-复制链接，用浏览器打开链接，地址栏一般变为https://www.kuaishou.com/profile/xxxxxx/开头的，复制xxxxxx过来" /> </StackPanel> <StackPanel Margin="0,12,0,0" Orientation="Horizontal"> <ui:TextBlock Width="60" VerticalAlignment="Center" Text="cookie" /> <ui:TextBox x:Name="tbCookie" Width="660" Height="36" VerticalContentAlignment="Center" ToolTip="利用浏览器开发者工具，从网络-请求标头中获取" /> </StackPanel> <StackPanel Margin="0,12,0,0" HorizontalAlignment="Center" Orientation="Horizontal"> <ui:Button x:Name="btnDownload" Height="32" Appearance="Primary" Click="Download_Click" Content="开始下载" CornerRadius="4 0 0 4" ToolTip="默认下载到程序根目录下，文件日期为作品发布日期" /> <ui:Button x:Name="btnParseJson" Height="32" Appearance="Primary" Click="ParseJson_Click" Content="..." CornerRadius="0 4 4 0" ToolTip="解析从web或者postman保存的json数据" /> </StackPanel> <TextBlock Width="700" Margin="0,12,0,0" Foreground="Gray" MouseDown="CopyUrl" Text="被快手风控不要慌，浏览器打开快手网页版，扫码登陆，点击我复制网址，粘贴到浏览器打开。打开后如果有很长很长的json数据返回，就对了。复制json保存到本地json文件，然后用第二个按钮解析json数据即可下载。" TextWrapping="Wrap" /> <Expander Margin="0,12,0,0" Header="更多选项"> <StackPanel Orientation="Horizontal"> <CheckBox x:Name="cbAddDate" Margin="12,0,0,0" VerticalAlignment="Center" Content="文件名前加上日期" IsChecked="True" ToolTip="文件名前面加上类似2024-01-02 13-00-00的标识，方便排序" /> <CheckBox x:Name="cbLongInterval" Margin="12,0,0,0" VerticalAlignment="Center" Content="增加作品下载延时" IsChecked="True" ToolTip="默认勾选，作品间下载延时5~10秒。取消勾选1~5秒随机，可能被风控" /> </StackPanel> </Expander> </StackPanel> <StackPanel Grid.Row="1" Margin="0,0,0,-2" VerticalAlignment="Bottom"> <TextBlock x:Name="tbProgress" HorizontalAlignment="Center" /> <ProgressBar x:Name="progress" Height="8" /> </StackPanel> <ui:Button x:Name="infoButton" Grid.Row="1" Width="32" Height="32" Margin="0,0,8,8" Padding="0" HorizontalAlignment="Right" VerticalAlignment="Bottom" Click="Info_Click" CornerRadius="16" FontSize="24" Icon="{ui:SymbolIcon Info28}" ToolTip="鸣谢" /> <ui:Flyout x:Name="flyout" Grid.Row="1" HorizontalAlignment="Right"> <ui:TextBlock Text="鸣谢: 
1. Microsoft Presentation Foundation
2. WPF-UI
3. RestSharp
4. Newtonsoft.Json
5. Downloader
6. Icon from FlatIcon" /> </ui:Flyout> </Grid></ui:FluentWindow>
- 后台逻辑没有使用MVVM，就是图方便。
点击查看代码
using KuaishouDownloader.Models;using Newtonsoft.Json;using RestSharp;using System.Diagnostics;using System.IO;using System.Text.RegularExpressions;using System.Windows;using Wpf.Ui;using Wpf.Ui.Controls;namespace KuaishouDownloader{ /// <summary> /// Interaction logic for MainWindow.xaml /// </summary> public partial class MainWindow { string downloadFolder = AppContext.BaseDirectory; SnackbarService? snackbarService = null; public MainWindow() { InitializeComponent(); this.Loaded += MainWindow_Loaded; } private void MainWindow_Loaded(object sender, RoutedEventArgs e) { snackbarService = new SnackbarService(); snackbarService.SetSnackbarPresenter(snackbarPresenter); if (File.Exists("AppConfig.json")) { var model = JsonConvert.DeserializeObject<AppConfig>(File.ReadAllText("AppConfig.json")); if (model != null) { tbUid.Text = model.Uid; tbCookie.Text = model.Cookie; } } } private void Theme_Click(object sender, RoutedEventArgs e) { if (Wpf.Ui.Appearance.ApplicationThemeManager.GetAppTheme() == Wpf.Ui.Appearance.ApplicationTheme.Light) { themeButton.Icon = new SymbolIcon(SymbolRegular.WeatherSunny48); Wpf.Ui.Appearance.ApplicationThemeManager.Apply(Wpf.Ui.Appearance.ApplicationTheme.Dark); } else { themeButton.Icon = new SymbolIcon(SymbolRegular.WeatherMoon48); Wpf.Ui.Appearance.ApplicationThemeManager.Apply(Wpf.Ui.Appearance.ApplicationTheme.Light); } } private async void Download_Click(object sender, RoutedEventArgs e) { try { btnDownload.IsEnabled = false; btnParseJson.IsEnabled = false; if (string.IsNullOrEmpty(tbUid.Text) || string.IsNullOrEmpty(tbCookie.Text)) { snackbarService?.Show("提示", $"请输入uid以及cookie", ControlAppearance.Caution, null, TimeSpan.FromSeconds(3)); return; } var json = JsonConvert.SerializeObject(new AppConfig() { Uid = tbUid.Text, Cookie = tbCookie.Text }, Formatting.Indented); File.WriteAllText("AppConfig.json", json); var options = new RestClientOptions("https://live.kuaishou.com") { Timeout = TimeSpan.FromSeconds(15), UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36", }; var client = new RestClient(options); var request = new RestRequest($"/live_api/profile/public?count=9999&pcursor=&principalId={tbUid.Text}&hasMore=true", Method.Get); request.AddHeader("host", "live.kuaishou.com"); request.AddHeader("connection", "keep-alive"); request.AddHeader("cache-control", "max-age=0"); request.AddHeader("sec-ch-ua", "\"Not)A;Brand\";v=\"99\", \"Google Chrome\";v=\"127\", \"Chromium\";v=\"127\""); request.AddHeader("sec-ch-ua-mobile", "?0"); request.AddHeader("sec-ch-ua-platform", "\"Windows\""); request.AddHeader("upgrade-insecure-requests", "1"); request.AddHeader("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"); request.AddHeader("sec-fetch-site", "none"); request.AddHeader("sec-fetch-mode", "navigate"); request.AddHeader("sec-fetch-user", "?1"); request.AddHeader("sec-fetch-dest", "document"); request.AddHeader("accept-encoding", "gzip, deflate, br, zstd"); request.AddHeader("accept-language", "zh,en;q=0.9,zh-CN;q=0.8"); request.AddHeader("cookie", tbCookie.Text); request.AddHeader("x-postman-captr", "9467712"); RestResponse response = await client.ExecuteAsync(request); Debug.WriteLine(response.Content); var model = JsonConvert.DeserializeObject<KuaishouModel>(response.Content!); if (model == null || model?.Data?.List == null || model?.Data?.List?.Count == 0) { snackbarService?.Show("提示", $"获取失败，可能触发了快手的风控机制，请等一段时间再试。", ControlAppearance.Danger, null, TimeSpan.FromSeconds(3)); return; } await Download(model!); } finally { btnDownload.IsEnabled = true; btnParseJson.IsEnabled = true; } } private async void ParseJson_Click(object sender, RoutedEventArgs e) { try { btnDownload.IsEnabled = false; btnParseJson.IsEnabled = false; var dialog = new Microsoft.Win32.OpenFileDialog(); dialog.Filter = "Json文件(.Json)|*.json"; bool? result = dialog.ShowDialog(); if (result == false) { return; } var model = JsonConvert.DeserializeObject<KuaishouModel>(File.ReadAllText(dialog.FileName)!); if (model == null || model?.Data?.List == null || model?.Data?.List?.Count == 0) { snackbarService?.Show("提示", $"不是正确的json", ControlAppearance.Caution, null, TimeSpan.FromSeconds(3)); return; } await Download(model!); } finally { btnDownload.IsEnabled = true; btnParseJson.IsEnabled = true; } } private async Task Download(KuaishouModel model) { progress.Value = 0; progress.Minimum = 0; progress.Maximum = (double)model?.Data?.List?.Count!; snackbarService?.Show("提示", $"解析到{model?.Data?.List?.Count!}个作品，开始下载", ControlAppearance.Success, null, TimeSpan.FromSeconds(5)); imgHeader.Source = new System.Windows.Media.Imaging.BitmapImage(new Uri(model?.Data?.List?[0]?.Author?.Avatar!)); tbNickName.Text = model?.Data?.List?[0]?.Author?.Name; string pattern = @"\d{4}/\d{2}/\d{2}/\d{2}"; for (int i = 0; i < model?.Data?.List!.Count; i++) { DateTime dateTime = DateTime.Now; string fileNamePrefix = ""; var item = model?.Data?.List[i]!; Match match = Regex.Match(item.Poster!, pattern); if (match.Success) { dateTime = new DateTime(int.Parse(match.Value.Split("/")[0]), int.Parse(match.Value.Split("/")[1]), int.Parse(match.Value.Split("/")[2]), int.Parse(match.Value.Split("/")[3]), 0, 0); if (cbAddDate.IsChecked == true) fileNamePrefix = match.Value.Split("/")[0] + "-" + match.Value.Split("/")[1] + "-" + match.Value.Split("/")[2] + " " + match.Value.Split("/")[3] + "-00-00 "; } downloadFolder = Path.Combine(AppContext.BaseDirectory, "Download", item?.Author?.Name! + "(" + item?.Author?.Id! + ")"); Directory.CreateDirectory(downloadFolder); switch (item?.WorkType) { case "single": case "vertical": case "multiple": { await DownLoadHelper.Download(item?.ImgUrls!, dateTime, downloadFolder, fileNamePrefix); } break; case "video": { await DownLoadHelper.Download(new List<string>() { item?.PlayUrl! }, dateTime, downloadFolder, fileNamePrefix); } break; } progress.Value = i + 1; tbProgress.Text = $"{i + 1} / {model?.Data?.List!.Count}"; Random random = new Random(); if (cbLongInterval.IsChecked == true) await Task.Delay(random.Next(5000, 10000)); else await Task.Delay(random.Next(1000, 5000)); } snackbarService?.Show("提示", $"下载完成，共下载{model?.Data?.List!.Count}个作品", ControlAppearance.Success, null, TimeSpan.FromDays(1)); } private void CopyUrl(object sender, System.Windows.Input.MouseButtonEventArgs e) { if (string.IsNullOrEmpty(tbUid.Text)) { snackbarService?.Show("提示", "请输入uid以及cookie", ControlAppearance.Caution, null, TimeSpan.FromSeconds(3)); return; } Clipboard.SetText($"https://live.kuaishou.com/live_api/profile/public?count=9999&pcursor=&principalId={tbUid.Text}&hasMore=true"); snackbarService?.Show("提示", "复制完成，请粘贴到浏览器打开", ControlAppearance.Success, null, TimeSpan.FromSeconds(3)); } private void Info_Click(object sender, RoutedEventArgs e) { flyout.IsOpen = true; } }}
- 下载类，下载完文件后，将文件的日志修改为发表日志，方便排序以及数据分析。
点击查看代码
public static async Task Download(List<string> urls, DateTime dateTime, string downloadFolder, string fileNamePrefix){ string file = string.Empty; try { var downloader = new DownloadService(); foreach (var url in urls) { Uri uri = new Uri(url); file = downloadFolder + "\\" + fileNamePrefix + Path.GetFileName(uri.LocalPath); if (!File.Exists(file)) await downloader.DownloadFileTaskAsync(url, file); //修改文件日期时间为发博的时间 File.SetCreationTime(file, dateTime); File.SetLastWriteTime(file, dateTime); File.SetLastAccessTime(file, dateTime); } } catch { Debug.WriteLine(file); Trace.Listeners.Add(new TextWriterTraceListener(downloadFolder + "\\_FailedFiles.txt", "myListener")); Trace.TraceInformation(file); Trace.Flush(); }}
- 源码分享
完整版代码已上传到Github https://github.com/hupo376787/KuaishouDownloader ，喜欢的点一下Star谢谢。
4. 下载使用
打开https://github.com/hupo376787/KuaishouDownloader/releases/tag/1.0，点击下载zip文件，解压缩后，就可以像开头那样使用了。
经验首页