: Shapley value regression / driver analysis with binary dependent variable. Does shapley support logistic regression models? Players cooperate in a coalition and receive a certain profit from this cooperation. Once all Shapley value shares are known, one may retrieve the coefficients (with original scale and origin) by solving an optimization problem suggested by Lipovetsky (2006) using any appropriate optimization method. Help comes from unexpected places: cooperative game theory. It is mind-blowing to explain a prediction as a game played by the feature values. LIME does not guarantee that the prediction is fairly distributed among the features. How Is the Partial Dependent Plot Calculated? In the identify causality series of articles, I demonstrate econometric techniques that identify causality. Shapley, Lloyd S. A value for n-person games. Contributions to the Theory of Games 2.28 (1953): 307-317., trumbelj, Erik, and Igor Kononenko. The driving forces identified by the KNN are: free sulfur dioxide, alcohol and residual sugar. It provides both global and local model-agnostic interpretation methods. The park-nearby contributed 30,000; area-50 contributed 10,000; floor-2nd contributed 0; cat-banned contributed -50,000. Not the answer you're looking for? Shapley computes feature contributions for single predictions with the Shapley value, an approach from cooperative game theory. The value floor-2nd was replaced by the randomly drawn floor-1st. The binary case is achieved in the notebook here. PMLR (2020)., Staniak, Mateusz, and Przemyslaw Biecek. Description. A data point close to the boundary means a low-confidence decision. The most common way of understanding a linear model is to examine the coefficients learned for each feature. Given the current set of feature values, the contribution of a feature value to the difference between the actual prediction and the mean prediction is the estimated Shapley value. Note that in the following algorithm, the order of features is not actually changed each feature remains at the same vector position when passed to the predict function. 1. Be Fluent in R and Python, Dimension Reduction Techniques with Python, Explain Any Models with the SHAP Values Use the KernelExplainer, https://sps.columbia.edu/faculty/chris-kuo. Lets build a random forest model and print out the variable importance. Mishra, S.K. I have also documented more recent development of the SHAP in The SHAP with More Elegant Charts and The SHAP Values with H2O Models. Head over to, \(x_o=(x_{(1)},\ldots,x_{(j)},\ldots,x_{(p)})\), \(z_o=(z_{(1)},\ldots,z_{(j)},\ldots,z_{(p)})\), \(x_{+j}=(x_{(1)},\ldots,x_{(j-1)},x_{(j)},z_{(j+1)},\ldots,z_{(p)})\), \(x_{-j}=(x_{(1)},\ldots,x_{(j-1)},z_{(j)},z_{(j+1)},\ldots,z_{(p)})\), \(\phi_j^{m}=\hat{f}(x_{+j})-\hat{f}(x_{-j})\), \(\phi_j(x)=\frac{1}{M}\sum_{m=1}^M\phi_j^{m}\), Output: Shapley value for the value of the j-th feature, Required: Number of iterations M, instance of interest x, feature index j, data matrix X, and machine learning model f, Draw random instance z from the data matrix X, Choose a random permutation o of the feature values. To learn more, see our tips on writing great answers. How to Increase accuracy and precision for my logistic regression model? Content Discovery initiative April 13 update: Related questions using a Review our technical responses for the 2023 Developer Survey. The SVM uses kernel functions to transform into a higher-dimensional space for the separation. There are two good papers to tell you a lot about the Shapley Value Regression: Lipovetsky, S. (2006). get_feature_names (), plot_type = 'dot') Explain the sentiment for one review I tried to follow the example notebook Github - SHAP: Sentiment Analysis with Logistic Regression but it seems it does not work as it is due to json . Results: Overall, 13,904 and 4259 individuals with prediabetes and diabetes, respectively, were identified in our underlying data set. It says mapping into a higher dimensional space often provides greater classification power. A solution for classification is logistic regression. The sum of contributions yields the difference between actual and average prediction (0.54). This is achieved by sampling values from the features marginal distribution. To learn more, see our tips on writing great answers. Site design / logo 2023 Stack Exchange Inc; user contributions licensed under CC BY-SA. Is it safe to publish research papers in cooperation with Russian academics? Shapley Regression. It does, but only if there are two classes. It shows the marginal effect that one or two variables have on the predicted outcome. The resulting values are no longer the Shapley values to our game, since they violate the symmetry axiom, as found out by Sundararajan et al. This is expected because we only train one SVM model and SVM is also prone to outliers. Regress (least squares) z on Qr to find R2q. Once it is obtained for each r, its arithmetic mean is computed. In situations where the law requires explainability like EUs right to explanations the Shapley value might be the only legally compliant method, because it is based on a solid theory and distributes the effects fairly. The temperature on this day had a positive contribution. The notebooks produced by AutoML regression and classification runs include code to calculate Shapley values. He also rips off an arm to use as a sword. Could a subterranean river or aquifer generate enough continuous momentum to power a waterwheel for the purpose of producing electricity? For features that appear left of the feature \(x_j\), we take the values from the original observations, and for the features on the right, we take the values from a random instance. The sum of Shapley values yields the difference of actual and average prediction (-2108). The SHAP values do not identify causality, which is better identified by experimental design or similar approaches. The interpretation of the Shapley value for feature value j is: In our apartment example, the feature values park-nearby, cat-banned, area-50 and floor-2nd worked together to achieve the prediction of 300,000. Because it makes not assumptions about the model type, KernelExplainer is slower than the other model type specific algorithms. background prior expectation for a home price \(E[f(X)]\), and then adds features one at a time until we reach the current model output \(f(x)\): The reason the partial dependence plots of linear models have such a close connection to SHAP values is because each feature in the model is handled independently of every other feature (the effects are just added together). Although the SHAP does not have built-in functions to save plots, you can output the plot by using matplotlib: The partial dependence plot, short for the dependence plot, is important in machine learning outcomes (J. H. Friedman 2001). Such additional scrutiny makes it practical to see how changes in the model impact results. It should be possible to choose M based on Chernoff bounds, but I have not seen any paper on doing this for Shapley values for machine learning predictions. I continue to produce the force plot for the 10th observation of the X_test data. Four powerful ML models were developed using data from male breast cancer (MBC) patients in the SEER database between 2010 and 2015 and . The logistic function is defined as: logistic() = 1 1 +exp() logistic ( ) = 1 1 + e x p ( ) And it looks like . Note that the bar plots above are just summary statistics from the values shown in the beeswarm plots below. The best answers are voted up and rise to the top, Not the answer you're looking for? The exponential number of the coalitions is dealt with by sampling coalitions and limiting the number of iterations M. Those articles cover the following techniques: Regression Discontinuity (see Identify Causality by Regression Discontinuity), Difference in differences (DiD)(see Identify Causality by Difference in Differences), Fixed-effects Models (See Identify Causality by Fixed-Effects Models), and Randomized Controlled Trial with Factorial Design (see Design of Experiments for Your Change Management). This property distinguishes the Shapley value from other methods such as LIME. Copyright 2018, Scott Lundberg. I assume in the regression case we do not know what the expected payoff is. rev2023.5.1.43405. For a game where a group of players cooperate, and where the expected payoff is known for each subset of players cooperating, one can calculate the Shapley value for each player, which is a way of fairly determining the contribution of each player to the payoff. (2016). Payout? This is because the value of each coefficient depends on the scale of the input features. The output shows that there is a linear and positive trend between alcohol and the target variable. xcolor: How to get the complementary color, Can corresponding author withdraw a paper after it has accepted without permission/acceptance of first author. If we are willing to deal with a bit more complexity we can use a beeswarm plot to summarize the entire distribution of SHAP values for each feature. The players are the feature values of the instance that collaborate to receive the gain (= predict a certain value). The Shapley value fairly distributes the difference of the instance's prediction and the datasets average prediction among the features. So if you have feedback or contributions please open an issue or pull request to make this tutorial better! Site design / logo 2023 Stack Exchange Inc; user contributions licensed under CC BY-SA. If, \[S\subseteq\{1,\ldots, p\} \backslash \{j,k\}\], Dummy The axioms efficiency, symmetry, dummy, additivity give the explanation a reasonable foundation. This is because a linear logistic regression model NOT additive in the probability space. Mathematically, the plot contains the following points: {(x ( i) j, ( i) j)}ni = 1. In Explain Your Model with the SHAP Values I use the function TreeExplainer() for a random forest model. rev2023.5.1.43405. I am not a lawyer, so this reflects only my intuition about the requirements. Connect and share knowledge within a single location that is structured and easy to search. Abstract and Figures. Instead, we model the payoff using some random variable and we have samples from this random variable. If all the force plots are combined, rotated 90 degrees, and stacked horizontally, we get the force plot of the entire data X_test (see the explanation of the GitHub of Lundberg and other contributors). My issue is that I want to be able to analyze a single prediction and get something more along these lines: In other words, I want to know which specific words contribute the most to the prediction. the Shapley value is the feature contribution to the prediction; It tells whether the relationship between the target and the variable is linear, monotonic, or more complex. This is done for all L combinations for a given r and arithmetic mean of Dr (over the sum of all L values of Dr) is computed. We draw r (r=0, 1, 2, , k-1) variables from Yi and let this collection of variables so drawn be called Pr such that Pr Yi . The SHAP values provide two great advantages: The SHAP values can be produced by the Python module SHAP. Are you Bilingual? The Shapley value is the wrong explanation method if you seek sparse explanations (explanations that contain few features). Very simply, the . The common kernel functions are Radial Basis Function (RBF), Gaussian, Polynomial, and Sigmoid. Binary outcome variables use logistic regression. Shapley additive explanation values were applied to select the important features. It only takes a minute to sign up. The effect of each feature is the weight of the feature times the feature value. A Medium publication sharing concepts, ideas and codes. Humans prefer selective explanations, such as those produced by LIME. Not the answer you're looking for? Continue exploring Asking for help, clarification, or responding to other answers. When we are explaining a prediction \(f(x)\), the SHAP value for a specific feature \(i\) is just the difference between the expected model output and the partial dependence plot at the features value \(x_i\): The close correspondence between the classic partial dependence plot and SHAP values means that if we plot the SHAP value for a specific feature across a whole dataset we will exactly trace out a mean centered version of the partial dependence plot for that feature: One of the fundemental properties of Shapley values is that they always sum up to the difference between the game outcome when all players are present and the game outcome when no players are present. Shapley Value: In game theory, a manner of fairly distributing both gains and costs to several actors working in coalition. The game is the prediction task for a single instance of the dataset. actually combines LIME implementation with Shapley values by using both the coefficients of a local . All clear now? Our goal is to explain the difference between the actual prediction (300,000) and the average prediction (310,000): a difference of -10,000. To subscribe to this RSS feed, copy and paste this URL into your RSS reader. features: HouseAge - median house age in block group, AveRooms - average number of rooms per household, AveBedrms - average number of bedrooms per household, AveOccup - average number of household members. To evaluate an existing model \(f\) when only a subset \(S\) of features are part of the model we integrate out the other features using a conditional expected value formulation. In this case, I suppose that you assume that the payoff is chi-squared? Relative Weights allows you to use as many variables as you want. But the mean absolute value is not the only way to create a global measure of feature importance, we can use any number of transforms. All these differences are averaged and result in: \[\phi_j(x)=\frac{1}{M}\sum_{m=1}^M\phi_j^{m}\]. I arbitrarily chose the 10th observation of the X_test data. I provide more detail in the article How Is the Partial Dependent Plot Calculated?. Note that Pr is null for r=0, and thus Qr contains a single variable, namely xi. Episode about a group who book passage on a space ship controlled by an AI, who turns out to be a human who can't leave his ship? Which language's style guidelines should be used when writing code that is supposed to be called from another language? Thanks, this was simpler than i though, i appreciate it. We will get better estimates if we repeat this sampling step and average the contributions. How to subdivide triangles into four triangles with Geometry Nodes? Thanks for contributing an answer to Cross Validated! The Shapley value is NOT the difference in prediction when we would remove the feature from the model. Clearly the number of years since a house Lundberg et al. Interpreting non-statistically significant results: Do we have "no evidence" or "insufficient evidence" to reject the null? By clicking Post Your Answer, you agree to our terms of service, privacy policy and cookie policy. A Support Vector Machine (AVM) finds the optimal hyperplane to separate observations into classes. Does the order of validations and MAC with clear text matter? LIME might be the better choice for explanations lay-persons have to deal with. This is an introduction to explaining machine learning models with Shapley values. The difference between the two R-squares is Dr = R2q - R2p, which is the marginal contribution of xi to z. Following this theory of sharing of the value of a game, the Shapley value regression decomposes the R2 (read it R square) of a conventional regression (which is considered as the value of the collusive cooperative game) such that the mean expected marginal contribution of every predictor variable (agents in collusion to explain the variation in y, the dependent variable) sums up to R2. I use his class H2OProbWrapper to calculate the SHAP values. It looks like you have just chosen an explainer that doesn't suit your model type. Here we show how using the max absolute value highights the Capital Gain and Capital Loss features, since they have infrewuent but high magnitude effects. Connect and share knowledge within a single location that is structured and easy to search. Another adaptation is conditional sampling: Features are sampled conditional on the features that are already in the team. The many Shapley values for model explanation. arXiv preprint arXiv:1908.08474 (2019)., Janzing, Dominik, Lenon Minorics, and Patrick Blbaum. By clicking Post Your Answer, you agree to our terms of service, privacy policy and cookie policy. Journal of Economics Bibliography, 3(3), 498-515. Be Fluent in R and Python in which I compare the most common data wrangling tasks in R dply and Python Pandas. summary_plot (shap_values [0], X_test_array, feature_names = vectorizer. While there are many ways to train these types of models (like setting an XGBoost model to depth-1), we will The feature importance for linear models in the presence of multicollinearity is known as the Shapley regression value or Shapley value13. Players? where \(\hat{f}(x^{m}_{+j})\) is the prediction for x, but with a random number of feature values replaced by feature values from a random data point z, except for the respective value of feature j. Why the obscure but specific description of Jane Doe II in the original complaint for Westenbroek v. Kappa Kappa Gamma Fraternity? 566), Improving the copy in the close modal and post notices - 2023 edition, New blog post from our CEO Prashanth: Community is the future of AI. Mobile Price Classification Interpreting Logistic Regression using SHAP Notebook Input Output Logs Comments (0) Run 343.7 s history Version 2 of 2 License This Notebook has been released under the Apache 2.0 open source license. It is interesting to mention a few R packages for the SHAP values here. Shapley value computes the regression using all possible combinations of predictors and computes the R 2 for each model. Why does Acts not mention the deaths of Peter and Paul? Before using Shapley values to explain complicated models, it is helpful to understand how they work for simple models. Suppose we want to get the dependence plot of alcohol. With a predicted 2409 rental bikes, this day is -2108 below the average prediction of 4518. ', referring to the nuclear power plant in Ignalina, mean? By clicking Accept all cookies, you agree Stack Exchange can store cookies on your device and disclose information in accordance with our Cookie Policy. Shapley values applied to a conditional expectation function of a machine learning model. The prediction for this observation is 5.00 which is similar to that of GBM. The exponential growth in the time needed to run Shapley regression places a constraint on the number of predictor variables that can be included in a model. Browse other questions tagged, Start here for a quick overview of the site, Detailed answers to any questions you might have, Discuss the workings and policies of this site. The Shapley value applies primarily in situations when the contributions . A unified approach to interpreting model predictions. Advances in Neural Information Processing Systems (2017)., Sundararajan, Mukund, and Amir Najmi. If your model is a deep learning model, use the deep learning explainer DeepExplainer(). All in all, the following coalitions are possible: For each of these coalitions we compute the predicted apartment price with and without the feature value cat-banned and take the difference to get the marginal contribution. Since I published the article Explain Your Model with the SHAP Values which was built on a random forest tree, readers have been asking if there is a universal SHAP Explainer for any ML algorithm either tree-based or non-tree-based algorithms. There are two options: one-vs-rest (ovr) or one-vs-one (ovo) (see the scikit-learn api). The drawback of the KernelExplainer is its long running time. Making statements based on opinion; back them up with references or personal experience. With a prediction of 0.57, this womans cancer probability is 0.54 above the average prediction of 0.03. These coefficients tell us how much the model output changes when we change each of the input features: While coefficients are great for telling us what will happen when we change the value of an input feature, by themselves they are not a great way to measure the overall importance of a feature. . All interpretable models explained in this book are interpretable on a modular level, with the exception of the k-nearest neighbors method. SHAP, an alternative estimation method for Shapley values, is presented in the next chapter. It is often crucial that the machine learning models are interpretable. This departure is expected because KNN is prone to outliers and here we only train a KNN model. The function KernelExplainer() below performs a local regression by taking the prediction method rf.predict and the data that you want to perform the SHAP values. Moreover, a SHAP value greater than zero leads to an increase in probability, a value less than zero leads to a decrease in probability. The result is the arithmetic average of the mean (or expected) marginal contributions of xi to z. \[\sum\nolimits_{j=1}^p\phi_j=\hat{f}(x)-E_X(\hat{f}(X))\], Symmetry The SHAP values look like this: SHAP values, first 5 passengers The higher the SHAP value the higher the probability of survival and vice versa. The Shapley value is defined via a value function \(val\) of players in S. The Shapley value of a feature value is its contribution to the payout, weighted and summed over all possible feature value combinations: \[\phi_j(val)=\sum_{S\subseteq\{1,\ldots,p\} \backslash \{j\}}\frac{|S|!\left(p-|S|-1\right)!}{p!}\left(val\left(S\cup\{j\}\right)-val(S)\right)\]. SHAP specifies the explanation as: $$\begin{aligned} f(x) = g\left( z^\prime \right) = \phi _0 + \sum \limits . Finally, the R package DALEX (Descriptive mAchine Learning EXplanations) also contains various explainers that help to understand the link between input variables and model output. The dependence plot of GBM also shows that there is an approximately linear and positive trend between alcohol and the target variable. It also lists other interpretable models. This formulation can take two I can see how this works for regression. Below are the average values of X_test, and the values of the 10th observation. Why did DOS-based Windows require HIMEM.SYS to boot? This section goes deeper into the definition and computation of the Shapley value for the curious reader. Which reverse polarity protection is better and why? Should I re-do this cinched PEX connection? The feature value is the numerical or categorical value of a feature and instance; What is the connection to machine learning predictions and interpretability? In Julia, you can use Shapley.jl. Find centralized, trusted content and collaborate around the technologies you use most. The R package xgboost has a built-in function. A higher-than-the-average sulfur dioxide (= 18 > 14.98) pushes the prediction to the right. In the post, I will demonstrate how to use the KernelExplainer for models built in KNN, SVM, Random Forest, GBM, or the H2O module. For a game with combined payouts val+val+ the respective Shapley values are as follows: Suppose you trained a random forest, which means that the prediction is an average of many decision trees. Machine learning is a powerful technology for products, research and automation. I was unable to find a solution with SHAP, but I found a solution using LIME. This looks similar to the feature contributions in the linear model! I am indebted to seanPLeary who has contributed to the H2O community on how to produce the SHAP values with AutoML. For more than a few features, the exact solution to this problem becomes problematic as the number of possible coalitions exponentially increases as more features are added. Why does the narrative change back and forth between "Isabella" and "Mrs. John Knightley" to refer to Emma's sister? Another solution comes from cooperative game theory: as an introduction to the shap Python package. Note that the blue partial dependence plot line (which the is average value of the model output when we fix the median income feature to a given value) always passes through the interesection of the two gray expected value lines. Each of these M new instances is a kind of Frankensteins Monster assembled from two instances. Parabolic, suborbital and ballistic trajectories all follow elliptic paths. What's the cheapest way to buy out a sibling's share of our parents house if I have no cash and want to pay less than the appraised value? The SHAP Python module does not yet have specifically optimized algorithms for all types of algorithms (such as KNNs). The average prediction for all apartments is 310,000. I also wrote a computer program (in Fortran 77) for Shapely regression. Part VI: An Explanation for eXplainable AI, Part V: Explain Any Models with the SHAP Values Use the KernelExplainer, Part VIII: Explain Your Model with Microsofts InterpretML. In the following figure we evaluate the contribution of the cat-banned feature value when it is added to a coalition of park-nearby and area-50. Follow More from Medium Aditya Bhattacharya in Towards Data Science Essential Explainable AI Python frameworks that you should know about Ani Madurkar in Towards Data Science Revision 45b85c18. where S is a subset of the features used in the model, x is the vector of feature values of the instance to be explained and p the number of features. It looks dotty because it is made of all the dots in the train data. The feature values of a data instance act as players in a coalition. It's not them. Thats exactly what the KernelExplainer, a model-agnostic method, is designed to do. By giving the features a new order, we get a random mechanism that helps us put together the Frankensteins Monster. In contrast to the output of the random forest, the SVM shows that alcohol interacts with fixed acidity frequently. Install Are these quarters notes or just eighth notes? Here I use the test dataset X_test which has 160 observations. The answer could be: A boy can regenerate, so demons eat him for years. For readers who want to get deeper into Machine Learning algorithms, you can check my post My Lecture Notes on Random Forest, Gradient Boosting, Regularization, and H2O.ai. Shapley values: a game theory approach Advantages & disadvantages The iml package is probably the most robust ML interpretability package available. Consider this question: Is your sophisticated machine-learning model easy to understand? That means your model can be understood by input variables that make business sense. SHAP (SHapley Additive exPlanations) is a game theoretic approach to explain the output of any machine learning model. Ulrike Grmping is the author of a R package called relaimpo in this package, she named this method which is based on this work lmg that calculates the relative importance when the predictor unlike the common methods has a relevant, known ordering. The scheme of Shapley value regression is simple. Use the SHAP Values to Interpret Your Sophisticated Model. When features are dependent, then we might sample feature values that do not make sense for this instance. The output of the KNN shows that there is an approximately linear and positive trend between alcohol and the target variable. My guess would go along these lines. If we sum all the feature contributions for one instance, the result is the following: \[\begin{align*}\sum_{j=1}^{p}\phi_j(\hat{f})=&\sum_{j=1}^p(\beta_{j}x_j-E(\beta_{j}X_{j}))\\=&(\beta_0+\sum_{j=1}^p\beta_{j}x_j)-(\beta_0+\sum_{j=1}^{p}E(\beta_{j}X_{j}))\\=&\hat{f}(x)-E(\hat{f}(X))\end{align*}\]. 3) Done. We are interested in how each feature affects the prediction of a data point. If you want to get more background on the SHAP values, I strongly recommend Explain Your Model with the SHAP Values, in which I describe carefully how the SHAP values emerge from the Shapley value, what the Shapley value in Game Theory, and how the SHAP values work in Python. Its principal application is to resolve a weakness of linear regression, which is that it is not reliable when predicted variables are moderately to highly correlated. How are engines numbered on Starship and Super Heavy? The feature contributions must add up to the difference of prediction for x and the average. To let you compare the results, I will use the same data source but use the function KernelExplainer(). The Dataman articles are my reflections on data science and teaching notes at Columbia University https://sps.columbia.edu/faculty/chris-kuo, rf = RandomForestRegressor(max_depth=6, random_state=0, n_estimators=10), shap.summary_plot(rf_shap_values, X_test), shap.dependence_plot("alcohol", rf_shap_values, X_test), # plot the SHAP values for the 10th observation, shap.force_plot(rf_explainer.expected_value, rf_shap_values, X_test), shap.summary_plot(gbm_shap_values, X_test), shap.dependence_plot("alcohol", gbm_shap_values, X_test), shap.force_plot(gbm_explainer.expected_value, gbm_shap_values, X_test), shap.summary_plot(knn_shap_values, X_test), shap.dependence_plot("alcohol", knn_shap_values, X_test), shap.force_plot(knn_explainer.expected_value, knn_shap_values, X_test), shap.summary_plot(svm_shap_values, X_test), shap.dependence_plot("alcohol", svm_shap_values, X_test), shap.force_plot(svm_explainer.expected_value, svm_shap_values, X_test), X_train, X_test = train_test_split(df, test_size = 0.1), X_test = X_test_hex.drop('quality').as_data_frame(), h2o_wrapper = H2OProbWrapper(h2o_rf,X_names), h2o_rf_explainer = shap.KernelExplainer(h2o_wrapper.predict_binary_prob, X_test), shap.summary_plot(h2o_rf_shap_values, X_test), shap.dependence_plot("alcohol", h2o_rf_shap_values, X_test), shap.force_plot(h2o_rf_explainer.expected_value, h2o_rf_shap_values, X_test), Explain Your Model with Microsofts InterpretML, My Lecture Notes on Random Forest, Gradient Boosting, Regularization, and H2O.ai, Explaining Deep Learning in a Regression-Friendly Way, A Technical Guide on RNN/LSTM/GRU for Stock Price Prediction, A unified approach to interpreting model predictions, Identify Causality by Regression Discontinuity, Identify Causality by Difference in Differences, Identify Causality by Fixed-Effects Models, Design of Experiments for Your Change Management.
Coaching Conversation Transcript,
How To Unsubmit An Assignment On Blackboard As A Student,
Biochemical Tests For Food Macromolecules,
Articles S
